2024-03-11T22:07:54.804 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:07:54.804 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985 -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/orig.config.yaml 2024-03-11T22:07:54.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:07:54.929 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591985' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuWNmKhPR/5LHxsOOVBDRBoJL4b7C7EW2Szf0NaC7gJjhoB2Ts3xlfV5fD3c4bByL89icpJDfD6bbQgjQ6tLyk= smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA60YmM6yAyWy2wrFD9iNyQnzN1QDkJ3w+ZnMysKY8OJdLSEEOIGwEX1EGRVyvHw9BEHp5HI5DJ22I7HtKvj3W8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-03-11T22:07:54.929 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:07:54.930 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:07:54.930 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:07:54.931 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:07:54.953 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:07:54.953 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:07:54.953 INFO:teuthology.packaging:ref: None 2024-03-11T22:07:54.953 INFO:teuthology.packaging:tag: None 2024-03-11T22:07:54.953 INFO:teuthology.packaging:branch: squid 2024-03-11T22:07:54.953 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:07:54.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:07:55.104 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:07:55.105 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:07:55.111 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:07:55.111 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:07:55.134 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:07:55.147 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:07:55.156 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:07:55.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:01:53.865377', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuWNmKhPR/5LHxsOOVBDRBoJL4b7C7EW2Szf0NaC7gJjhoB2Ts3xlfV5fD3c4bByL89icpJDfD6bbQgjQ6tLyk='} 2024-03-11T22:07:55.200 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:01:53.866952', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA60YmM6yAyWy2wrFD9iNyQnzN1QDkJ3w+ZnMysKY8OJdLSEEOIGwEX1EGRVyvHw9BEHp5HI5DJ22I7HtKvj3W8='} 2024-03-11T22:07:55.200 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:07:55.208 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-11T22:07:55.208 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-11T22:07:55.208 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:07:55.277 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff0ae127f70>, signals=[15]) 2024-03-11T22:07:55.278 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:07:55.288 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:07:55.288 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:07:55.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:07:55.379 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-03-11T22:07:55.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:07:55.467 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:07:55.474 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-03-11T22:07:55.492 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-03-11T22:07:55.492 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:07:55.549 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:07:55.550 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-03-11T22:07:55.573 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-03-11T22:07:55.591 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-03-11T22:07:55.591 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="9" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="9" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:07:55.648 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:07:55.649 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:07:55.649 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:07:55.649 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:07:55.649 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:07:55.649 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-03-11T22:07:55.673 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:07:55.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:07:55.692 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:07:55.692 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:07:55.694 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:07:55.710 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:07:55.717 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:07:55.717 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:07:55.750 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:07:55.851 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:07:55.905 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:07:55.921 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:07:55.921 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:07:55.922 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:07:55.922 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:07:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:07:55.939 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-03-11T22:07:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:07:55.940 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-11T22:07:55.996 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:07:55.996 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-361.el9.x86_64 2024-03-11T22:07:55.996 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2024-03-11T22:07:55.997 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:07:55.997 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.14.0-361.el9.x86_64 2024-03-11T22:07:55.997 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-03-11T22:07:58.134 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:07:58.134 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:07:59.585 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-11T22:07:59.592 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-11T22:08:01.075 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-11T22:08:01.159 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-11T22:08:03.884 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-11T22:08:09.226 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-11T22:08:10.760 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-11T22:08:13.620 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-11T22:08:17.073 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-11T22:08:18.503 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-11T22:08:20.659 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:08:20.660 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:08:20.752 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-03-11T22:08:20.758 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2024-03-11T22:08:20.759 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:08:20.760 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 100 M 2024-03-11T22:08:20.760 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 123 M 2024-03-11T22:08:20.760 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-03-11T22:08:21.821 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 6.8 MB/s | 6.0 MB 00:00 2024-03-11T22:08:22.924 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-11T22:08:23.321 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 8.8 MB/s | 21 MB 00:02 2024-03-11T22:08:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 13 MB/s | 39 MB 00:03 2024-03-11T22:08:24.571 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 12 MB/s | 33 MB 00:02 2024-03-11T22:08:24.571 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:08:24.571 INFO:teuthology.orchestra.run.smithi073.stdout:Total 26 MB/s | 100 MB 00:03 2024-03-11T22:08:24.607 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-03-11T22:08:25.453 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-03-11T22:08:25.834 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-03-11T22:08:25.834 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-03-11T22:08:27.184 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-03-11T22:08:27.185 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-03-11T22:08:27.675 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-11T22:08:29.900 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-03-11T22:08:30.543 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:08:30.680 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:08:32.608 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-11T22:08:33.473 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:08:33.648 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:08:34.766 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:08:34.766 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:08:34.847 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-11T22:08:34.854 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-11T22:08:34.854 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-03-11T22:08:34.854 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-11T22:08:34.854 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-11T22:08:34.854 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout:Install 4 Packages 2024-03-11T22:08:34.855 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:08:34.856 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 100 M 2024-03-11T22:08:34.856 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 123 M 2024-03-11T22:08:34.856 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-11T22:08:36.993 INFO:teuthology.orchestra.run.smithi099.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 2.9 MB/s | 6.0 MB 00:02 2024-03-11T22:08:37.651 INFO:teuthology.orchestra.run.smithi099.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.6 MB/s | 21 MB 00:02 2024-03-11T22:08:38.373 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:08:38.499 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:38.526 INFO:teuthology.orchestra.run.smithi099.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 22 MB/s | 33 MB 00:01 2024-03-11T22:08:42.367 INFO:teuthology.orchestra.run.smithi099.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.3 MB/s | 39 MB 00:07 2024-03-11T22:08:42.370 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:08:42.370 INFO:teuthology.orchestra.run.smithi099.stdout:Total 13 MB/s | 100 MB 00:07 2024-03-11T22:08:43.234 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:08:43.261 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-11T22:08:43.642 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-11T22:08:43.642 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-11T22:08:45.267 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-11T22:08:45.268 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-11T22:08:47.978 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-11T22:08:48.631 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:08:48.777 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:08:51.567 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:08:51.756 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:08:56.483 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:08:56.617 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:01.361 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:12.508 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:12.509 INFO:teuthology.orchestra.run.smithi073.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:09:12.509 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:09:12.594 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:13.067 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:13.068 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:09:13.068 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:09:13.068 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:09:13.498 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:09:13.499 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-03-11T22:09:14.020 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:09:14.164 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:14.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:09:14.165 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:09:14.165 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:09:14.165 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:09:14.165 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-11T22:09:14.165 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:09:14.165 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:09:14.285 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:14.286 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:09:14.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.06 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 68.el9 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Group : Unspecified 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 8264254 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:09:14.323 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : builder@centos.org 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:09:14.324 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:09:14.325 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:09:14.325 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:09:14.325 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:09:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2024-03-11T22:09:16.320 INFO:teuthology.orchestra.run.smithi073.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:09:16.354 INFO:teuthology.orchestra.run.smithi073.stderr:done 2024-03-11T22:09:16.357 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-03-11T22:09:16.372 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.bl22pRTlgy 2024-03-11T22:09:16.372 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bl22pRTlgy 2024-03-11T22:09:16.441 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.bl22pRTlgy 2024-03-11T22:09:16.585 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.bl22pRTlgy is 6KB 2024-03-11T22:09:16.604 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.bl22pRTlgy 2024-03-11T22:09:16.618 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:09:16.682 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:09:16.683 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:09:16.683 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:09:16.683 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:09:16.684 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:09:16.830 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2024-03-11T22:09:30.479 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:30.479 INFO:teuthology.orchestra.run.smithi099.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:09:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:09:30.562 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:09:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:09:31.018 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:09:31.454 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:09:31.455 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-11T22:09:31.971 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:09:32.181 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:32.181 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:09:32.181 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:09:32.181 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:09:32.181 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:09:32.182 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-11T22:09:32.182 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:09:32.182 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:09:32.302 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:09:32.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:09:32.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.06 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 68.el9 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Group : Unspecified 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 8264254 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:09:32.343 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : builder@centos.org 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:09:32.344 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:09:32.345 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:09:32.345 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:09:32.345 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:09:32.955 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-03-11T22:09:34.344 INFO:teuthology.orchestra.run.smithi099.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:09:34.384 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-03-11T22:09:34.387 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-11T22:09:34.401 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.6bTBYRqXvg 2024-03-11T22:09:34.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6bTBYRqXvg 2024-03-11T22:09:34.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.6bTBYRqXvg 2024-03-11T22:09:34.607 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.6bTBYRqXvg is 6KB 2024-03-11T22:09:34.655 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.6bTBYRqXvg 2024-03-11T22:09:34.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:09:34.734 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:09:34.734 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:09:34.734 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:09:34.734 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:09:34.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:09:34.886 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-03-11T22:09:46.845 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-11T22:09:46.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:09:46.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:10:04.903 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-11T22:10:04.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:10:04.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:10:05.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-03-11T22:10:14.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:10:14.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:10:17.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-03-11T22:10:23.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-11T22:10:29.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:10:29.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:10:32.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:10:32.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:11:01.154 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-03-11T22:11:01.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:11:01.619 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:11:01.619 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-03-11T22:11:01.633 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:11:01.633 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:11:01.633 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:11:01.633 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:11:02.634 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-11T22:11:02.635 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:11:02.727 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2024-03-11T22:11:02.768 DEBUG:teuthology.parallel:result is None 2024-03-11T22:11:32.397 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:11:44.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:11:44.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:11:44.816 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-11T22:11:45.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:11:45.245 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:11:45.245 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-11T22:11:45.261 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:11:45.261 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:11:45.262 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:11:45.262 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:11:46.262 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-11T22:11:46.263 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:11:46.342 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-03-11T22:11:46.386 DEBUG:teuthology.parallel:result is None 2024-03-11T22:11:46.386 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:11:46.394 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:11:46.394 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:11:46.397 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:11:46.415 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:11:46.422 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:11:46.430 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:11:46.430 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:11:46.456 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:11:46.486 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:11:46.493 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:11:46.493 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:11:46.524 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:11:46.557 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:11:46.563 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:11:46.571 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:11:46.576 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:11:46.578 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:11:46.585 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:11:46.585 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:11:46.618 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:11:46.647 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:11:46.657 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:11:46.657 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:11:46.686 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:11:46.703 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:11:46.782 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:11:46.831 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:11:46.916 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:11:46.957 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:11:46.957 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:11:47.024 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:11:47.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:11:47.129 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:11:47.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:11:47.214 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:11:47.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:11:47.282 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-03-11T22:11:47.284 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-03-11T22:11:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:11:47.367 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:11:47.745 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:11:47.753 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:11:47.753 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:11:47.783 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:11:47.793 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T22:11:47.793 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2024-03-11T22:11:47.862 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2024-03-11T22:11:47.864 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-03-11T22:11:47.927 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-03-11T22:11:47.929 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:11:47.929 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2024-03-11T22:11:47.953 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2024-03-11T22:11:47.953 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-03-11T22:11:47.971 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-03-11T22:11:47.971 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi073.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi099.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:11:47.971 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:11:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:11:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.051 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:11:48.051 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:11:48.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:11:48.131 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:11:48.140 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:11:48.157 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:11:48.159 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com,smithi099.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:16:02.946 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com'), Remote(name='ubuntu@smithi099.front.sepia.ceph.com')] 2024-03-11T22:16:02.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:16:02.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:16:03.039 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-03-11T22:16:03.112 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-11T22:16:03.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:16:03.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:16:03.202 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-11T22:16:03.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-11T22:16:03.282 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:16:03.292 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:16:03.292 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:16:03.293 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:16:03.295 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:16:03.295 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:16:03.323 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:16:03.339 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:16:03.353 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:16:03.368 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2024-03-11T22:16:03.383 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:16:03.383 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2024-03-11T22:16:03.398 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:16:03.399 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-03-11T22:16:03.413 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-03-11T22:16:03.416 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:16:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:16:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:16:03.469 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:16:03.473 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:16:03.473 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-03-11T22:16:03.473 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.474 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.474 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.474 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.497 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:16:03.502 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:16:03.596 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:16:03.596 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-11T22:16:03.596 INFO:tasks.cephadm:Cluster fsid is f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:03.596 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:16:03.597 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-11T22:16:03.597 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi073': '172.21.15.73', 'mon.smithi099': '172.21.15.99'} 2024-03-11T22:16:03.597 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:16:03.597 DEBUG:teuthology.orchestra.run.smithi073:> sudo hostname $(hostname -s) 2024-03-11T22:16:03.624 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-03-11T22:16:03.652 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-11T22:16:03.653 DEBUG:teuthology.orchestra.run.smithi073:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:03.936 INFO:teuthology.orchestra.run.smithi073.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 11 22:16 /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:03.936 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:04.114 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 11 22:16 /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:04.114 DEBUG:teuthology.orchestra.run.smithi073:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:04.130 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:16:04.155 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-11T22:16:04.155 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-11T22:16:04.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-11T22:16:05.069 INFO:teuthology.orchestra.run.smithi073.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T22:16:05.081 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout: "repo_digests": [ 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-11T22:16:29.388 INFO:teuthology.orchestra.run.smithi073.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-11T22:16:29.389 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-11T22:16:29.389 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:16:32.210 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "repo_digests": [ 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-11T22:16:32.211 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-11T22:16:32.229 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph 2024-03-11T22:16:32.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-03-11T22:16:32.294 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /etc/ceph 2024-03-11T22:16:32.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-03-11T22:16:32.362 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:16:32.363 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:16:32.364 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T22:16:32.364 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:16:32.364 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:16:32.393 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f2f121a8-dff4-11ee-95c8-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T22:16:32.393 DEBUG:teuthology.orchestra.run.smithi073:mon.smithi073> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service 2024-03-11T22:16:32.436 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:16:32.436 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.73 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:16:32.633 INFO:teuthology.orchestra.run.smithi073.stderr:-------------------------------------------------------------------------------- 2024-03-11T22:16:32.633 INFO:teuthology.orchestra.run.smithi073.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'f2f121a8-dff4-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.73', '--skip-admin-label'] 2024-03-11T22:16:32.655 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: 4.9.3-dev 2024-03-11T22:16:32.657 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying podman|docker is present... 2024-03-11T22:16:32.678 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: 4.9.3-dev 2024-03-11T22:16:32.681 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying lvm2 is present... 2024-03-11T22:16:32.681 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying time synchronization is in place... 2024-03-11T22:16:32.687 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:16:32.693 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:16:32.700 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: enabled 2024-03-11T22:16:32.706 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: active 2024-03-11T22:16:32.707 INFO:teuthology.orchestra.run.smithi073.stderr:Unit chronyd.service is enabled and running 2024-03-11T22:16:32.707 INFO:teuthology.orchestra.run.smithi073.stderr:Repeating the final host check... 2024-03-11T22:16:32.728 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: 4.9.3-dev 2024-03-11T22:16:32.730 INFO:teuthology.orchestra.run.smithi073.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:16:32.730 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl is present 2024-03-11T22:16:32.730 INFO:teuthology.orchestra.run.smithi073.stderr:lvcreate is present 2024-03-11T22:16:32.735 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:16:32.742 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:16:32.748 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: enabled 2024-03-11T22:16:32.754 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: active 2024-03-11T22:16:32.755 INFO:teuthology.orchestra.run.smithi073.stderr:Unit chronyd.service is enabled and running 2024-03-11T22:16:32.755 INFO:teuthology.orchestra.run.smithi073.stderr:Host looks OK 2024-03-11T22:16:32.755 INFO:teuthology.orchestra.run.smithi073.stderr:Cluster fsid: f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:32.755 INFO:teuthology.orchestra.run.smithi073.stderr:Acquiring lock 140403855826320 on /run/cephadm/f2f121a8-dff4-11ee-95c8-87774f69a715.lock 2024-03-11T22:16:32.755 INFO:teuthology.orchestra.run.smithi073.stderr:Lock 140403855826320 acquired on /run/cephadm/f2f121a8-dff4-11ee-95c8-87774f69a715.lock 2024-03-11T22:16:32.756 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying IP 172.21.15.73 port 3300 ... 2024-03-11T22:16:32.756 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying IP 172.21.15.73 port 6789 ... 2024-03-11T22:16:32.756 INFO:teuthology.orchestra.run.smithi073.stderr:Base mon IP is 172.21.15.73, final addrv is [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-03-11T22:16:32.758 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.73 metric 100 2024-03-11T22:16:32.758 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.73 metric 100 2024-03-11T22:16:32.761 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:16:32.761 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T22:16:32.761 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1760sec hoplimit 64 pref medium 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-11T22:16:32.764 INFO:teuthology.orchestra.run.smithi073.stderr:Mon IP `172.21.15.73` is in CIDR network `172.21.0.0/20` 2024-03-11T22:16:32.765 INFO:teuthology.orchestra.run.smithi073.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:16:32.765 INFO:teuthology.orchestra.run.smithi073.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-11T22:16:32.866 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-11T22:16:33.355 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Getting image source signatures 2024-03-11T22:16:33.355 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-11T22:16:33.356 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-11T22:16:33.356 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-11T22:16:33.356 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-11T22:16:33.356 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-11T22:16:33.356 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-11T22:16:33.524 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: Writing manifest to image destination 2024-03-11T22:16:33.674 INFO:teuthology.orchestra.run.smithi073.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-11T22:16:34.047 INFO:teuthology.orchestra.run.smithi073.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-11T22:16:34.744 INFO:teuthology.orchestra.run.smithi073.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-11T22:16:34.744 INFO:teuthology.orchestra.run.smithi073.stderr:Extracting ceph user uid/gid from container image... 2024-03-11T22:16:35.151 INFO:teuthology.orchestra.run.smithi073.stderr:stat: 167 167 2024-03-11T22:16:35.665 INFO:teuthology.orchestra.run.smithi073.stderr:Creating initial keys... 2024-03-11T22:16:36.056 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph-authtool: AQDDgu9l2J31ORAAS6X//7hKRoQ9U3l4kHT//Q== 2024-03-11T22:16:36.903 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph-authtool: AQDEgu9lL27UMBAArB55meDybvXf9r0hyoSjZg== 2024-03-11T22:16:37.742 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph-authtool: AQDFgu9lyaK2JhAAmgDBxN69QNdpfPxhjkyEXg== 2024-03-11T22:16:43.241 INFO:teuthology.orchestra.run.smithi073.stderr:Creating initial monmap... 2024-03-11T22:16:46.986 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:16:46.986 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-11T22:16:46.986 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:46.986 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:16:49.708 INFO:teuthology.orchestra.run.smithi073.stderr:monmaptool for smithi073 [v2:172.21.15.73:3300,v1:172.21.15.73:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:16:49.708 INFO:teuthology.orchestra.run.smithi073.stderr:setting min_mon_release = octopus 2024-03-11T22:16:49.709 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: set fsid to f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:49.709 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:16:49.709 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:16:49.709 INFO:teuthology.orchestra.run.smithi073.stderr:Creating mon... 2024-03-11T22:16:50.722 INFO:teuthology.orchestra.run.smithi073.stderr:create mon.smithi073 on 2024-03-11T22:16:50.945 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:16:51.091 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target. 2024-03-11T22:16:51.091 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target. 2024-03-11T22:16:51.385 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Failed to reset failed state of unit ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service: Unit ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service not loaded. 2024-03-11T22:16:51.393 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target.wants/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service → /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@.service. 2024-03-11T22:16:52.068 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:52 smithi073 podman[24417]: 2024-03-11 22:16:52.003022673 +0000 UTC m=+0.345845335 container start 1e34e655196146efc1609a724b236cafdafcfb881f11225efcd696c1906d3db0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_CLEAN=True, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, name=centos-stream) 2024-03-11T22:16:52.068 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:52 smithi073 ceph-mon[24435]: mkfs f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:52.108 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: disabled 2024-03-11T22:16:52.116 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:16:52.117 INFO:teuthology.orchestra.run.smithi073.stderr:firewalld.service is not enabled 2024-03-11T22:16:52.117 INFO:teuthology.orchestra.run.smithi073.stderr:Not possible to enable service . firewalld.service is not available 2024-03-11T22:16:52.118 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mon to start... 2024-03-11T22:16:52.118 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mon... 2024-03-11T22:16:52.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:52 smithi073 ceph-mon[24435]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-03-11T22:16:52.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:52 smithi073 bash[24417]: 1e34e655196146efc1609a724b236cafdafcfb881f11225efcd696c1906d3db0 2024-03-11T22:16:52.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:52 smithi073 systemd[1]: Started Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: cluster: 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: id: f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: services: 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi073 (age 0.567744s) 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:52.633 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: data: 2024-03-11T22:16:52.634 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-11T22:16:52.634 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-11T22:16:52.634 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:16:52.634 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: pgs: 2024-03-11T22:16:52.634 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:53.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-03-11T22:16:53.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: monmap e1: 1 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} 2024-03-11T22:16:53.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: fsmap 2024-03-11T22:16:53.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:16:53.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: mgrmap e1: no daemons active 2024-03-11T22:16:53.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:53 smithi073 ceph-mon[24435]: from='client.? 172.21.15.73:0/1660229712' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:16:53.235 INFO:teuthology.orchestra.run.smithi073.stderr:mon is available 2024-03-11T22:16:53.235 INFO:teuthology.orchestra.run.smithi073.stderr:Assimilating anything we can from ceph.conf... 2024-03-11T22:16:53.696 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:53.696 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [global] 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: fsid = f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [mgr] 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [osd] 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-11T22:16:53.697 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-11T22:16:54.261 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:54 smithi073 ceph-mon[24435]: from='client.? 172.21.15.73:0/1196422440' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:16:54.262 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:54 smithi073 ceph-mon[24435]: from='client.? 172.21.15.73:0/1196422440' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:Generating new minimal ceph.conf... 2024-03-11T22:16:55.462 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:55 smithi073 ceph-mon[24435]: from='client.? 172.21.15.73:0/2360332952' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:16:56.936 INFO:teuthology.orchestra.run.smithi073.stderr:Restarting the monitor... 2024-03-11T22:16:57.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:56 smithi073 systemd[1]: Stopping Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:16:57.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:56 smithi073 bash[24665]: Error: no container with name or ID "ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon.smithi073" found: no such container 2024-03-11T22:16:57.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[24431]: 2024-03-11T22:16:57.103+0000 7fbbe0ca0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi073 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:16:57.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[24431]: 2024-03-11T22:16:57.103+0000 7fbbe0ca0700 -1 mon.smithi073@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:16:57.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 podman[24678]: 2024-03-11 22:16:57.247452984 +0000 UTC m=+0.251489150 container died 1e34e655196146efc1609a724b236cafdafcfb881f11225efcd696c1906d3db0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., version=8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T22:16:57.762 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 podman[24678]: 2024-03-11 22:16:57.586752249 +0000 UTC m=+0.590788411 container cleanup 1e34e655196146efc1609a724b236cafdafcfb881f11225efcd696c1906d3db0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-03-11T22:16:57.762 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 bash[24678]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073 2024-03-11T22:16:58.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 podman[24691]: 2024-03-11 22:16:57.762622129 +0000 UTC m=+0.514833679 container remove 1e34e655196146efc1609a724b236cafdafcfb881f11225efcd696c1906d3db0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, ceph=True, io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-03-11T22:16:58.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 bash[24704]: Error: no container with name or ID "ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon.smithi073" found: no such container 2024-03-11T22:16:58.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service: Deactivated successfully. 2024-03-11T22:16:58.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 systemd[1]: Stopped Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:58.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:57 smithi073 systemd[1]: Starting Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 podman[24775]: 2024-03-11 22:16:58.030034614 +0000 UTC m=+0.111230616 container create 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, io.openshift.expose-services=, architecture=x86_64, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, name=centos-stream, io.buildah.version=1.19.8) 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 podman[24775]: 2024-03-11 22:16:57.938108092 +0000 UTC m=+0.019304108 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 podman[24775]: 2024-03-11 22:16:58.232656018 +0000 UTC m=+0.313852028 container init 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, version=8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.display-name=CentOS Stream 8) 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 podman[24775]: 2024-03-11 22:16:58.235532903 +0000 UTC m=+0.316728905 container start 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, maintainer=Guillaume Abrioux , ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, release=754, version=8, distribution-scope=public, architecture=x86_64, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: pidfile_write: ignore empty --pid-file 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: load: jerasure load: lrc 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: RocksDB version: 6.15.5 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T22:16:58.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Compile date Apr 18 2022 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: DB SUMMARY 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: DB Session ID: 0YJ099TNW8DYLICI5IQI 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi073/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi073/store.db: 000010.log size: 83216 ; 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.env: 0x55d6c58c6860 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.fs: Posix File System 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.info_log: 0x55d6c7da3fa0 2024-03-11T22:16:58.312 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.statistics: (nil) 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.use_fsync: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:16:58.313 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.db_log_dir: 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi073/store.db 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.write_buffer_manager: 0x55d6c7e94330 2024-03-11T22:16:58.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.unordered_write: 0 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:16:58.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.row_cache: None 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.wal_filter: None 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.preserve_deletes: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:16:58.316 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_open_files: -1 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:16:58.317 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Compression algorithms supported: 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kZSTD supported: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:16:58.318 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000009 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.merge_operator: 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_filter: None 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d6c7d6fd00) 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_type: 0 2024-03-11T22:16:58.319 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_index_type: 0 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_shortening: 1 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: hash_index_allow_collision: 1 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: checksum: 1 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: no_block_cache: 0 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache: 0x55d6c7dda170 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_options: 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: capacity : 536870912 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: num_shard_bits : 4 2024-03-11T22:16:58.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: strict_capacity_limit : 0 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_compressed: (nil) 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: persistent_cache: (nil) 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size: 4096 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size_deviation: 10 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_restart_interval: 16 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_block_restart_interval: 1 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: metadata_block_size: 4096 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: partition_filters: 0 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: use_delta_encoding: 1 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: whole_key_filtering: 1 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: verify_compression: 0 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: format_version: 4 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_index_compression: 1 2024-03-11T22:16:58.321 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_align: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression: NoCompression 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.num_levels: 7 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:16:58.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:16:58.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:16:58.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:16:58.325 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.ttl: 2592000 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:16:58.326 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195418262075, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195418263090, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79654, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77918, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9900, "raw_average_key_size": 47, "raw_value_size": 72214, "raw_average_value_size": 343, "num_data_blocks": 10, "num_entries": 210, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710195418, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "5ebe29d6-3596-4d09-8fe2-0918ea937098", "db_session_id": "0YJ099TNW8DYLICI5IQI"}} 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195418263504, "job": 1, "event": "recovery_finished"} 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi073/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:16:58.327 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d6c7dc0a80 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: DB pointer 0x55d6c7dd0000 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** DB Stats ** 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:16:58.328 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L0 2/0 79.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Sum 2/0 79.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:16:58.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative compaction: 0.00 GB write, 18.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval compaction: 0.00 GB write, 18.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L0 2/0 79.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Sum 2/0 79.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:16:58.330 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 112.5 0.00 0.00 1 0.001 0 0 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative compaction: 0.00 GB write, 18.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: starting mon.smithi073 rank 0 at public addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] at bind addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi073 fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:58.331 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???) e1 preinit fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).mds e1 new map 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).mds e1 print_map 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: e1 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: legacy client fscid: -1 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: No filesystems configured 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:16:58.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 bash[24775]: 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 2024-03-11T22:16:58.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 systemd[1]: Started Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:16:58.333 INFO:teuthology.orchestra.run.smithi073.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-11T22:16:58.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mon.smithi073 is new leader, mons smithi073 in quorum (ranks 0) 2024-03-11T22:16:58.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: monmap e1: 1 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} 2024-03-11T22:16:58.576 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: fsmap 2024-03-11T22:16:58.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:16:58.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:58 smithi073 ceph-mon[24794]: mgrmap e1: no daemons active 2024-03-11T22:16:59.257 INFO:teuthology.orchestra.run.smithi073.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:16:59.257 INFO:teuthology.orchestra.run.smithi073.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:16:59.257 INFO:teuthology.orchestra.run.smithi073.stderr:Creating mgr... 2024-03-11T22:16:59.257 INFO:teuthology.orchestra.run.smithi073.stderr:Verifying port 9283 ... 2024-03-11T22:16:59.429 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Failed to reset failed state of unit ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mgr.smithi073.pglkvm.service: Unit ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mgr.smithi073.pglkvm.service not loaded. 2024-03-11T22:16:59.435 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715.target.wants/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mgr.smithi073.pglkvm.service → /etc/systemd/system/ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@.service. 2024-03-11T22:16:59.942 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:16:59 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3590062360' entity='client.admin' 2024-03-11T22:17:00.152 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: disabled 2024-03-11T22:17:00.159 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:17:00.160 INFO:teuthology.orchestra.run.smithi073.stderr:firewalld.service is not enabled 2024-03-11T22:17:00.160 INFO:teuthology.orchestra.run.smithi073.stderr:Not possible to enable service . firewalld.service is not available 2024-03-11T22:17:00.168 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: disabled 2024-03-11T22:17:00.175 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:17:00.176 INFO:teuthology.orchestra.run.smithi073.stderr:firewalld.service is not enabled 2024-03-11T22:17:00.176 INFO:teuthology.orchestra.run.smithi073.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-11T22:17:00.176 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mgr to start... 2024-03-11T22:17:00.176 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mgr... 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsid": "f2f121a8-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "health": { 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T22:17:00.666 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 0 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "smithi073" 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T22:17:00.667 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T22:17:00.668 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T22:17:00.672 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "available": false, 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T22:17:00.673 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "iostat", 2024-03-11T22:17:00.674 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "nfs", 2024-03-11T22:17:00.674 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "restful" 2024-03-11T22:17:00.674 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:00.674 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:00.674 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modified": "2024-03-11T22:16:52.065227+0000", 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T22:17:00.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:00.955 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:00 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3070612479' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:17:01.258 INFO:teuthology.orchestra.run.smithi073.stderr:mgr not available, waiting (1/15)... 2024-03-11T22:17:03.795 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:03.795 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:03.795 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsid": "f2f121a8-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:17:03.795 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "health": { 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 0 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "smithi073" 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T22:17:03.797 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T22:17:03.801 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "available": false, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "iostat", 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "nfs", 2024-03-11T22:17:03.802 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "restful" 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modified": "2024-03-11T22:16:52.065227+0000", 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T22:17:03.803 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:04.397 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:04 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2457112337' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:17:04.442 INFO:teuthology.orchestra.run.smithi073.stderr:mgr not available, waiting (2/15)... 2024-03-11T22:17:06.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: mgrmap e2: smithi073.pglkvm(active, starting, since 0.00295137s) 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:05 smithi073 ceph-mon[24794]: from='mgr.14100 172.21.15.73:0/315212151' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:07.138 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsid": "f2f121a8-dff4-11ee-95c8-87774f69a715", 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "health": { 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "checks": {}, 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mutes": [] 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum": [ 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 0 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-11T22:17:07.139 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "smithi073" 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "monmap": { 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osdmap": { 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-11T22:17:07.140 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgmap": { 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-11T22:17:07.141 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "fsmap": { 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "available": true, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modules": [ 2024-03-11T22:17:07.142 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "iostat", 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "nfs", 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "restful" 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ], 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "servicemap": { 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "modified": "2024-03-11T22:16:52.065227+0000", 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "services": {} 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: }, 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-11T22:17:07.143 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:07.721 INFO:teuthology.orchestra.run.smithi073.stderr:mgr is available 2024-03-11T22:17:07.861 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:07 smithi073 ceph-mon[24794]: mgrmap e3: smithi073.pglkvm(active, since 1.00622s) 2024-03-11T22:17:07.861 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:07 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/754730626' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [global] 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: fsid = f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:08.325 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [mgr] 2024-03-11T22:17:08.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-11T22:17:08.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 2024-03-11T22:17:08.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: [osd] 2024-03-11T22:17:08.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-11T22:17:08.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-11T22:17:08.871 INFO:teuthology.orchestra.run.smithi073.stderr:Enabling cephadm module... 2024-03-11T22:17:09.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:09 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1667490430' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:17:09.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:09 smithi073 ceph-mon[24794]: mgrmap e4: smithi073.pglkvm(active, since 3s) 2024-03-11T22:17:10.591 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:10 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2732236876' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:17:11.462 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:11 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2732236876' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:17:11.462 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:11 smithi073 ceph-mon[24794]: mgrmap e5: smithi073.pglkvm(active, since 4s) 2024-03-11T22:17:11.557 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:11.558 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-11T22:17:11.558 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "available": true, 2024-03-11T22:17:11.558 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "active_name": "smithi073.pglkvm", 2024-03-11T22:17:11.558 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-11T22:17:11.558 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:12.176 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for the mgr to restart... 2024-03-11T22:17:12.176 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mgr epoch 5... 2024-03-11T22:17:12.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:12 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3006853846' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:17:16.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:16 smithi073 ceph-mon[24794]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:17:16.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:16 smithi073 ceph-mon[24794]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:17:16.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:16 smithi073 ceph-mon[24794]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:17:17.110 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:17.110 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-11T22:17:17.110 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "initialized": true 2024-03-11T22:17:17.110 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:17.420 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: mgrmap e6: smithi073.pglkvm(active, starting, since 0.0523826s) 2024-03-11T22:17:17.420 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:17:17.420 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:17:17.420 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:17:17.420 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:17.421 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:17 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:17.673 INFO:teuthology.orchestra.run.smithi073.stderr:mgr epoch 5 is available 2024-03-11T22:17:17.674 INFO:teuthology.orchestra.run.smithi073.stderr:Setting orchestrator backend to cephadm... 2024-03-11T22:17:18.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:16] ENGINE Bus STARTING 2024-03-11T22:17:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:16] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:17:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:16] ENGINE Bus STARTED 2024-03-11T22:17:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:17:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: mgrmap e7: smithi073.pglkvm(active, since 1.05723s) 2024-03-11T22:17:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:18 smithi073 ceph-mon[24794]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:17:19.428 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:19 smithi073 ceph-mon[24794]: mgrmap e8: smithi073.pglkvm(active, since 2s) 2024-03-11T22:17:19.428 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:19 smithi073 ceph-mon[24794]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:19.428 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:19 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:19.428 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:19 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:19.429 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: value unchanged 2024-03-11T22:17:19.969 INFO:teuthology.orchestra.run.smithi073.stderr:Generating ssh key... 2024-03-11T22:17:20.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:20 smithi073 ceph-mon[24794]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:22.032 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:21 smithi073 ceph-mon[24794]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:22.032 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:21 smithi073 ceph-mon[24794]: Generating ssh key... 2024-03-11T22:17:22.033 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:21 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:22.033 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:21 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:22.033 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrHHFVC6lds/LO/ArDgJYWHa3+qM1+1tujKWgRnltk2B9eNLert6dw6QEsmF2WqQ03xv47jIRE88xi4aB7lN6ThpcNGY6HlaZsvcGdlpWJR0szGlvfOu4GS9Gi37pEjqViXxBvgbp9bDn7gHxizQR1Y7cLRUe8Y1/7i/R5kP0wNbYoAPnq5EYKQ2vAY8ArgeyJEUSMFUMrxUFPS9xx4DbH9oBiuanK5+UU+TO+905qQm9tFpbGqBwu8cyESahcoKg4XtUaV6pMyzIc+QvuuAwUPTjDJpup8YW2Be2Aj7zzCx5rsdWxUtdIAOLz9RdJTryxIudUw7NZ32/SERGZaAyf+8R5Fqaupf9v0bCzhc5d4zmeqGnuGn9+KpQU+IdL5TlBfIp5B8zmTaaLtE1jxVwJuId93JlbrQwKJ1hPj5lWsHdPKpCVEw9tDNWNIbwoZersO+52uCeYtkkCnnXnVv/8h1TpicDVqTVwyzLyNQ56OUBqrjdmSh0fnwcbGPInGRE= ceph-f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:22.648 INFO:teuthology.orchestra.run.smithi073.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:17:22.648 INFO:teuthology.orchestra.run.smithi073.stderr:Adding key to root@localhost authorized_keys... 2024-03-11T22:17:22.648 INFO:teuthology.orchestra.run.smithi073.stderr:Adding host smithi073... 2024-03-11T22:17:23.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:22 smithi073 ceph-mon[24794]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:24.260 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Added host 'smithi073' with addr '172.21.15.73' 2024-03-11T22:17:24.586 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:24 smithi073 ceph-mon[24794]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi073", "addr": "172.21.15.73", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:24.586 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:24 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:24.586 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:24 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:24.863 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying mon service with default placement... 2024-03-11T22:17:25.457 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:25 smithi073 ceph-mon[24794]: Deploying cephadm binary to smithi073 2024-03-11T22:17:25.457 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:25 smithi073 ceph-mon[24794]: Added host smithi073 2024-03-11T22:17:25.457 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-11T22:17:25.928 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying mgr service with default placement... 2024-03-11T22:17:26.712 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:26 smithi073 ceph-mon[24794]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:26.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:26 smithi073 ceph-mon[24794]: Saving service mon spec with placement count:5 2024-03-11T22:17:26.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:26 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-11T22:17:27.381 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying crash service with default placement... 2024-03-11T22:17:28.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:27 smithi073 ceph-mon[24794]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:28.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:27 smithi073 ceph-mon[24794]: Saving service mgr spec with placement count:2 2024-03-11T22:17:28.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:27 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:28.349 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled crash update... 2024-03-11T22:17:29.151 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying prometheus service with default placement... 2024-03-11T22:17:29.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:29 smithi073 ceph-mon[24794]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:29.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:29 smithi073 ceph-mon[24794]: Saving service crash spec with placement * 2024-03-11T22:17:29.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:29 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:29.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:29 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:29.795 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-03-11T22:17:30.525 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying grafana service with default placement... 2024-03-11T22:17:30.548 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:30 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:30.549 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:30 smithi073 ceph-mon[24794]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:30.549 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:30 smithi073 ceph-mon[24794]: Saving service prometheus spec with placement count:1 2024-03-11T22:17:30.549 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:30 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:31.128 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-03-11T22:17:31.741 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying node-exporter service with default placement... 2024-03-11T22:17:32.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:32 smithi073 ceph-mon[24794]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:32.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:32 smithi073 ceph-mon[24794]: Saving service grafana spec with placement count:1 2024-03-11T22:17:32.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:32 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:32.326 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-03-11T22:17:32.948 INFO:teuthology.orchestra.run.smithi073.stderr:Deploying alertmanager service with default placement... 2024-03-11T22:17:33.342 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:33 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:33.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:33 smithi073 ceph-mon[24794]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:33.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:33 smithi073 ceph-mon[24794]: Saving service node-exporter spec with placement * 2024-03-11T22:17:33.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:33 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:33.675 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-03-11T22:17:34.631 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:34 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:34.631 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:34 smithi073 ceph-mon[24794]: from='mgr.14118 172.21.15.73:0/3544748585' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:35.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:35 smithi073 ceph-mon[24794]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:35.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:35 smithi073 ceph-mon[24794]: Saving service alertmanager spec with placement count:1 2024-03-11T22:17:36.908 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:36 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2024374002' entity='client.admin' 2024-03-11T22:17:36.908 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:36 smithi073 ceph-mon[24794]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:37.508 INFO:teuthology.orchestra.run.smithi073.stderr:Enabling the dashboard module... 2024-03-11T22:17:38.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3552351223' entity='client.admin' 2024-03-11T22:17:39.212 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:38 smithi073 ceph-mon[24794]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:17:39.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:38 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1192488814' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:17:40.109 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:39 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1192488814' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:17:40.109 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:39 smithi073 ceph-mon[24794]: mgrmap e9: smithi073.pglkvm(active, since 22s) 2024-03-11T22:17:40.288 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:40.288 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-11T22:17:40.288 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "available": true, 2024-03-11T22:17:40.289 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "active_name": "smithi073.pglkvm", 2024-03-11T22:17:40.289 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-11T22:17:40.289 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:40.922 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for the mgr to restart... 2024-03-11T22:17:40.922 INFO:teuthology.orchestra.run.smithi073.stderr:Waiting for mgr epoch 9... 2024-03-11T22:17:40.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:40 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2844377089' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:17:45.134 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:44 smithi073 ceph-mon[24794]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:17:45.134 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:44 smithi073 ceph-mon[24794]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:17:45.134 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:44 smithi073 ceph-mon[24794]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:17:45.895 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: { 2024-03-11T22:17:45.895 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-11T22:17:45.895 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: "initialized": true 2024-03-11T22:17:45.896 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: } 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: mgrmap e10: smithi073.pglkvm(active, starting, since 0.055868s) 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:17:46.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:46.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:46.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:17:46.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:17:46.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:45 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:46.649 INFO:teuthology.orchestra.run.smithi073.stderr:mgr epoch 9 is available 2024-03-11T22:17:46.649 INFO:teuthology.orchestra.run.smithi073.stderr:Generating a dashboard self-signed certificate... 2024-03-11T22:17:46.900 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:45] ENGINE Bus STARTING 2024-03-11T22:17:46.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:45] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:17:46.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: [11/Mar/2024:22:17:45] ENGINE Bus STARTED 2024-03-11T22:17:46.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:17:46.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: mgrmap e11: smithi073.pglkvm(active, since 1.06349s) 2024-03-11T22:17:46.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:46 smithi073 ceph-mon[24794]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:17:47.379 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-11T22:17:47.989 INFO:teuthology.orchestra.run.smithi073.stderr:Creating initial admin user... 2024-03-11T22:17:48.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:47 smithi073 ceph-mon[24794]: mgrmap e12: smithi073.pglkvm(active, since 2s) 2024-03-11T22:17:48.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:47 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:48.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:47 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:48.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:47 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:48.872 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$b6lYTNNTM4aZ9AsWyTJha.tyjrkOVHAFLVmakLRIFNvdgwdh5s5K6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710195468, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:17:49.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:48 smithi073 ceph-mon[24794]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:49.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:48 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:49.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:48 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:49.498 INFO:teuthology.orchestra.run.smithi073.stderr:Fetching dashboard port number... 2024-03-11T22:17:50.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:49 smithi073 ceph-mon[24794]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:17:51.006 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: 8443 2024-03-11T22:17:51.576 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: disabled 2024-03-11T22:17:51.583 INFO:teuthology.orchestra.run.smithi073.stderr:systemctl: inactive 2024-03-11T22:17:51.583 INFO:teuthology.orchestra.run.smithi073.stderr:firewalld.service is not enabled 2024-03-11T22:17:51.584 INFO:teuthology.orchestra.run.smithi073.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr:Ceph Dashboard is now available at: 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr: URL: https://smithi073.front.sepia.ceph.com:8443/ 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr: User: admin 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr: Password: pxlmwz1h18 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:51.586 INFO:teuthology.orchestra.run.smithi073.stderr:Enabling autotune for osd_memory_target 2024-03-11T22:17:52.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:51 smithi073 ceph-mon[24794]: mgrmap e13: smithi073.pglkvm(active, since 6s) 2024-03-11T22:17:52.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:51 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3571996973' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:17:53.581 INFO:teuthology.orchestra.run.smithi073.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-11T22:17:54.197 INFO:teuthology.orchestra.run.smithi073.stderr:You can access the Ceph CLI with: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: ceph telemetry on 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr:For more information see: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:17:54.198 INFO:teuthology.orchestra.run.smithi073.stderr:Bootstrap complete. 2024-03-11T22:17:54.236 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:17:54.236 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:17:54.236 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:17:54.252 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:17:54.252 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:17:54.252 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:17:54.307 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:17:54.307 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:17:54.307 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/keyring of=/dev/stdout 2024-03-11T22:17:54.373 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:17:54.374 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:17:54.374 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:17:54.428 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:17:54.428 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrHHFVC6lds/LO/ArDgJYWHa3+qM1+1tujKWgRnltk2B9eNLert6dw6QEsmF2WqQ03xv47jIRE88xi4aB7lN6ThpcNGY6HlaZsvcGdlpWJR0szGlvfOu4GS9Gi37pEjqViXxBvgbp9bDn7gHxizQR1Y7cLRUe8Y1/7i/R5kP0wNbYoAPnq5EYKQ2vAY8ArgeyJEUSMFUMrxUFPS9xx4DbH9oBiuanK5+UU+TO+905qQm9tFpbGqBwu8cyESahcoKg4XtUaV6pMyzIc+QvuuAwUPTjDJpup8YW2Be2Aj7zzCx5rsdWxUtdIAOLz9RdJTryxIudUw7NZ32/SERGZaAyf+8R5Fqaupf9v0bCzhc5d4zmeqGnuGn9+KpQU+IdL5TlBfIp5B8zmTaaLtE1jxVwJuId93JlbrQwKJ1hPj5lWsHdPKpCVEw9tDNWNIbwoZersO+52uCeYtkkCnnXnVv/8h1TpicDVqTVwyzLyNQ56OUBqrjdmSh0fnwcbGPInGRE= ceph-f2f121a8-dff4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:17:54.506 INFO:teuthology.orchestra.run.smithi073.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrHHFVC6lds/LO/ArDgJYWHa3+qM1+1tujKWgRnltk2B9eNLert6dw6QEsmF2WqQ03xv47jIRE88xi4aB7lN6ThpcNGY6HlaZsvcGdlpWJR0szGlvfOu4GS9Gi37pEjqViXxBvgbp9bDn7gHxizQR1Y7cLRUe8Y1/7i/R5kP0wNbYoAPnq5EYKQ2vAY8ArgeyJEUSMFUMrxUFPS9xx4DbH9oBiuanK5+UU+TO+905qQm9tFpbGqBwu8cyESahcoKg4XtUaV6pMyzIc+QvuuAwUPTjDJpup8YW2Be2Aj7zzCx5rsdWxUtdIAOLz9RdJTryxIudUw7NZ32/SERGZaAyf+8R5Fqaupf9v0bCzhc5d4zmeqGnuGn9+KpQU+IdL5TlBfIp5B8zmTaaLtE1jxVwJuId93JlbrQwKJ1hPj5lWsHdPKpCVEw9tDNWNIbwoZersO+52uCeYtkkCnnXnVv/8h1TpicDVqTVwyzLyNQ56OUBqrjdmSh0fnwcbGPInGRE= ceph-f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:54.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrHHFVC6lds/LO/ArDgJYWHa3+qM1+1tujKWgRnltk2B9eNLert6dw6QEsmF2WqQ03xv47jIRE88xi4aB7lN6ThpcNGY6HlaZsvcGdlpWJR0szGlvfOu4GS9Gi37pEjqViXxBvgbp9bDn7gHxizQR1Y7cLRUe8Y1/7i/R5kP0wNbYoAPnq5EYKQ2vAY8ArgeyJEUSMFUMrxUFPS9xx4DbH9oBiuanK5+UU+TO+905qQm9tFpbGqBwu8cyESahcoKg4XtUaV6pMyzIc+QvuuAwUPTjDJpup8YW2Be2Aj7zzCx5rsdWxUtdIAOLz9RdJTryxIudUw7NZ32/SERGZaAyf+8R5Fqaupf9v0bCzhc5d4zmeqGnuGn9+KpQU+IdL5TlBfIp5B8zmTaaLtE1jxVwJuId93JlbrQwKJ1hPj5lWsHdPKpCVEw9tDNWNIbwoZersO+52uCeYtkkCnnXnVv/8h1TpicDVqTVwyzLyNQ56OUBqrjdmSh0fnwcbGPInGRE= ceph-f2f121a8-dff4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:17:54.567 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrHHFVC6lds/LO/ArDgJYWHa3+qM1+1tujKWgRnltk2B9eNLert6dw6QEsmF2WqQ03xv47jIRE88xi4aB7lN6ThpcNGY6HlaZsvcGdlpWJR0szGlvfOu4GS9Gi37pEjqViXxBvgbp9bDn7gHxizQR1Y7cLRUe8Y1/7i/R5kP0wNbYoAPnq5EYKQ2vAY8ArgeyJEUSMFUMrxUFPS9xx4DbH9oBiuanK5+UU+TO+905qQm9tFpbGqBwu8cyESahcoKg4XtUaV6pMyzIc+QvuuAwUPTjDJpup8YW2Be2Aj7zzCx5rsdWxUtdIAOLz9RdJTryxIudUw7NZ32/SERGZaAyf+8R5Fqaupf9v0bCzhc5d4zmeqGnuGn9+KpQU+IdL5TlBfIp5B8zmTaaLtE1jxVwJuId93JlbrQwKJ1hPj5lWsHdPKpCVEw9tDNWNIbwoZersO+52uCeYtkkCnnXnVv/8h1TpicDVqTVwyzLyNQ56OUBqrjdmSh0fnwcbGPInGRE= ceph-f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:17:54.582 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:17:54.657 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:54 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1154333108' entity='client.admin' 2024-03-11T22:17:57.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:57 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:57.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:57 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:17:57.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:57 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:17:57.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:57 smithi073 ceph-mon[24794]: Deploying daemon alertmanager.smithi073 on smithi073 2024-03-11T22:17:57.959 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:17:57.959 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:17:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:17:58 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2636141758' entity='client.admin' 2024-03-11T22:18:01.302 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi099 2024-03-11T22:18:01.302 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:18:01.302 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:18:01.320 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:18:01.320 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:18:01.377 INFO:tasks.cephadm:Adding host smithi099 to orchestrator... 2024-03-11T22:18:01.378 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch host add smithi099 2024-03-11T22:18:01.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:01 smithi073 ceph-mon[24794]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:01.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:01 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:04.994 INFO:teuthology.orchestra.run.smithi073.stdout:Added host 'smithi099' with addr '172.21.15.99' 2024-03-11T22:18:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:05 smithi073 ceph-mon[24794]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:05 smithi073 ceph-mon[24794]: Deploying cephadm binary to smithi099 2024-03-11T22:18:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:05 smithi073 ceph-mon[24794]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:06.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:05 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:06.283 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:18:07.123 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:06 smithi073 ceph-mon[24794]: Added host smithi099 2024-03-11T22:18:08.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:07 smithi073 ceph-mon[24794]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:09.574 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:09 smithi073 ceph-mon[24794]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:10.332 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:18:10.332 INFO:teuthology.orchestra.run.smithi073.stdout:[{"addr": "172.21.15.73", "hostname": "smithi073", "labels": [], "status": ""}, {"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}] 2024-03-11T22:18:11.354 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:18:11.354 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:18:12.183 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:11 smithi073 ceph-mon[24794]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:18:12.184 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:11 smithi073 ceph-mon[24794]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: Deploying daemon crash.smithi073 on smithi073 2024-03-11T22:18:13.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:13 smithi073 ceph-mon[24794]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:14.286 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2024-03-11T22:18:14.364 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:14 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3128752278' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:18:14.932 INFO:tasks.cephadm:Adding mon.smithi073 on smithi073 2024-03-11T22:18:14.932 INFO:tasks.cephadm:Adding mon.smithi099 on smithi099 2024-03-11T22:18:14.932 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch apply mon '2;smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099' 2024-03-11T22:18:15.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:15 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3128752278' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:18:15.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:15 smithi073 ceph-mon[24794]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:18:15.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:15 smithi073 ceph-mon[24794]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:15.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:15 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:16.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:16.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:16.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:16.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:18:16.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:18:16.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:16.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:16 smithi073 ceph-mon[24794]: Deploying daemon grafana.smithi073 on smithi073 2024-03-11T22:18:16.834 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled mon update... 2024-03-11T22:18:17.438 DEBUG:teuthology.orchestra.run.smithi099:mon.smithi099> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi099.service 2024-03-11T22:18:17.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:17.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:17 smithi073 ceph-mon[24794]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:18:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:17 smithi073 ceph-mon[24794]: Saving service mon spec with placement smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099;count:2 2024-03-11T22:18:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:17 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:18.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:17 smithi073 ceph-mon[24794]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:19.410 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:19.411 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:19.412 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:19.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:19 smithi073 ceph-mon[24794]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:20.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:20 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/4115107722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:20.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:20 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:20.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:20.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:21.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:21 smithi073 ceph-mon[24794]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:22.904 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:22.905 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:22.907 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:23.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:22 smithi073 ceph-mon[24794]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:23.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:22 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/106984624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:24.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:24.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:26.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:25 smithi073 ceph-mon[24794]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:26.301 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:26.301 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:27.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:26 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/2854519415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:27.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:27.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:28.192 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:27 smithi073 ceph-mon[24794]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:29.642 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:29 smithi073 ceph-mon[24794]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:29.737 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:29.738 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:30.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:30 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/1844575035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:31.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:31.298 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:31.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:31 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:31.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:31 smithi073 ceph-mon[24794]: Deploying daemon node-exporter.smithi073 on smithi073 2024-03-11T22:18:31.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:31 smithi073 ceph-mon[24794]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:33.146 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:33.147 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:33.148 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:33.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:33 smithi073 ceph-mon[24794]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:18:33.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:33 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/2768463868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:33.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:33 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:33.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:33 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:18:34.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:34.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:36.051 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:35 smithi073 ceph-mon[24794]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:18:36.051 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:35 smithi073 ceph-mon[24794]: mgrmap e14: smithi073.pglkvm(active, since 49s) 2024-03-11T22:18:36.509 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:36.509 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:37.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:36 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3811132814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:38.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:38.088 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:39.903 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:39.903 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:39.905 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:40.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:39 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/724594454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:41.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:40 smithi073 ceph-mon[24794]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:18:41.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:40 smithi073 ceph-mon[24794]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:18:41.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:40 smithi073 ceph-mon[24794]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:18:41.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:41.424 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: mgrmap e15: smithi073.pglkvm(active, starting, since 0.308473s) 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:42.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: [11/Mar/2024:22:18:41] ENGINE Bus STARTING 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: [11/Mar/2024:22:18:41] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: [11/Mar/2024:22:18:41] ENGINE Bus STARTED 2024-03-11T22:18:42.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:43.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:42 smithi073 ceph-mon[24794]: mgrmap e16: smithi073.pglkvm(active, since 1.3147s) 2024-03-11T22:18:43.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:43.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:43.506 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:43.507 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:43.509 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:44.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:44 smithi073 ceph-mon[24794]: mgrmap e17: smithi073.pglkvm(active, since 2s) 2024-03-11T22:18:44.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:44 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/2676306524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:44.859 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:44 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:45.168 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:45.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:47.730 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:47.731 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:48.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:47 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/275833933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:49.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:49.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:49.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:49.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:49.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:50.213 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:49 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:50.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:49 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:50.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:49 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:18:50.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:49 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:18:50.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:49 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:18:51.258 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:51.259 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:51.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:18:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:18:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:51 smithi073 ceph-mon[24794]: Deploying daemon crash.smithi099 on smithi099 2024-03-11T22:18:52.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:52 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/2859617921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:53.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:53.600 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:54 smithi073 ceph-mon[24794]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:18:55.551 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:18:55.552 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:18:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:18:55.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:55 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/1477312868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:18:57.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:57 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:57.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:18:57.969 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:18:58.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:18:58.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:18:58.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:58.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:18:58 smithi073 ceph-mon[24794]: Deploying daemon mon.smithi099 on smithi099 2024-03-11T22:19:01.848 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:19:01.848 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:16:46.128529Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:19:01.850 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-03-11T22:19:01.953 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 podman[27104]: 2024-03-11 22:19:01.933255793 +0000 UTC m=+0.350401449 container start 803548aea8809419bb1964a1ec5b4ac979234f840a6df80644303bed96d539f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD) 2024-03-11T22:19:01.953 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:19:01.954 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-11T22:19:01.954 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pidfile_write: ignore empty --pid-file 2024-03-11T22:19:02.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: load: jerasure load: lrc 2024-03-11T22:19:02.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: RocksDB version: 6.15.5 2024-03-11T22:19:02.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T22:19:02.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Compile date Apr 18 2022 2024-03-11T22:19:02.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: DB SUMMARY 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: DB Session ID: 53K6PLMX0USK9O0MF9D9 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 0, files: 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 000004.log size: 511 ; 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.env: 0x564cbaac6860 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.fs: Posix File System 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.info_log: 0x564cbb4a5fa0 2024-03-11T22:19:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.statistics: (nil) 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.use_fsync: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.db_log_dir: 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi099/store.db 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:19:02.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.write_buffer_manager: 0x564cbb596330 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:19:02.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.unordered_write: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.row_cache: None 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.wal_filter: None 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.preserve_deletes: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:19:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:19:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:19:02.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:19:02.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:19:02.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_open_files: -1 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Compression algorithms supported: 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kZSTD supported: 0 2024-03-11T22:19:02.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000003 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.merge_operator: 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_filter: None 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564cbb471d00) 2024-03-11T22:19:02.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_type: 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: data_block_index_type: 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_shortening: 1 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: hash_index_allow_collision: 1 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: checksum: 1 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: no_block_cache: 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache: 0x564cbb4dc170 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_options: 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: capacity : 536870912 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: num_shard_bits : 4 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: strict_capacity_limit : 0 2024-03-11T22:19:02.247 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_compressed: (nil) 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: persistent_cache: (nil) 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_size: 4096 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_size_deviation: 10 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_restart_interval: 16 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_block_restart_interval: 1 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: metadata_block_size: 4096 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: partition_filters: 0 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: use_delta_encoding: 1 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: whole_key_filtering: 1 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: verify_compression: 0 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: format_version: 4 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: enable_index_compression: 1 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_align: 0 2024-03-11T22:19:02.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression: NoCompression 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.num_levels: 7 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:19:02.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:19:02.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:19:02.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:19:02.252 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.ttl: 2592000 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-11T22:19:02.253 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195541959204, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195541959500, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710195541, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3e188185-4663-4bac-902d-c0db1e698852", "db_session_id": "53K6PLMX0USK9O0MF9D9"}} 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710195541959807, "job": 1, "event": "recovery_finished"} 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi099/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x564cbb4c2a80 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: DB pointer 0x564cbb4d2000 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** DB Stats ** 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:19:02.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:19:02.255 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:19:02.256 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:19:02.257 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099 does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: using public_addr v2:172.21.15.99:0/0 -> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: starting mon.smithi099 rank -1 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi099 fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(???) e0 preinit fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).mds e1 new map 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).mds e1 print_map 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: e1 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: legacy client fscid: -1 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:19:02.258 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: No filesystems configured 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.259 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon crash.smithi073 on smithi073 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3128752278' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3128752278' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.260 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon grafana.smithi073 on smithi073 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Saving service mon spec with placement smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099;count:2 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/4115107722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/106984624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/2854519415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/1844575035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon node-exporter.smithi073 on smithi073 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/2768463868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14160 172.21.15.73:0/1752257182' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mgrmap e14: smithi073.pglkvm(active, since 49s) 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3811132814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/724594454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:19:02.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mgrmap e15: smithi073.pglkvm(active, starting, since 0.308473s) 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:19:02.263 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: [11/Mar/2024:22:18:41] ENGINE Bus STARTING 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: [11/Mar/2024:22:18:41] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: [11/Mar/2024:22:18:41] ENGINE Bus STARTED 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mgrmap e16: smithi073.pglkvm(active, since 1.3147s) 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mgrmap e17: smithi073.pglkvm(active, since 2s) 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/2676306524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/275833933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.264 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon crash.smithi099 on smithi099 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/2859617921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:02.265 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/1477312868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: Deploying daemon mon.smithi099 on smithi099 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:01 smithi099 ceph-mon[27157]: mon.smithi099@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:02 smithi099 bash[27104]: 803548aea8809419bb1964a1ec5b4ac979234f840a6df80644303bed96d539f0 2024-03-11T22:19:02.266 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:02 smithi099 systemd[1]: Started Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:19:03.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:19:03.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: mon.smithi073 calling monitor election 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: mon.smithi099 calling monitor election 2024-03-11T22:19:07.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: monmap e2: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: fsmap 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: mgrmap e17: smithi073.pglkvm(active, since 26s) 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: overall HEALTH_OK 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:19:07.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:07.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:19:07.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: mon.smithi073 calling monitor election 2024-03-11T22:19:07.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: mon.smithi099 calling monitor election 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: monmap e2: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: fsmap 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: mgrmap e17: smithi073.pglkvm(active, since 26s) 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:19:07.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:19:07.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: overall HEALTH_OK 2024-03-11T22:19:07.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:19:07.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/4278813268' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:19:07.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:08.442 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:08 smithi099 ceph-mon[27157]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:08.442 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:08 smithi099 ceph-mon[27157]: Deploying daemon node-exporter.smithi099 on smithi099 2024-03-11T22:19:08.442 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:08 smithi099 ceph-mon[27157]: mgrmap e18: smithi073.pglkvm(active, since 26s), standbys: smithi099.rcoixq 2024-03-11T22:19:08.442 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:19:08.442 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:08.442 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-11T22:19:08.442 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":2,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","modified":"2024-03-11T22:19:01.982400Z","created":"2024-03-11T22:16:46.128529Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi073","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:19:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 2 2024-03-11T22:19:08.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:08 smithi073 ceph-mon[24794]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:08.463 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:08 smithi073 ceph-mon[24794]: Deploying daemon node-exporter.smithi099 on smithi099 2024-03-11T22:19:08.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:08 smithi073 ceph-mon[24794]: mgrmap e18: smithi073.pglkvm(active, since 26s), standbys: smithi099.rcoixq 2024-03-11T22:19:08.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:19:08.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:19:09.202 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:19:09.202 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:19:09.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:09 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3611971506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:09.514 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:09 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3611971506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:19:10.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:10 smithi073 ceph-mon[24794]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:10.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:10 smithi099 ceph-mon[27157]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:11.015 INFO:teuthology.orchestra.run.smithi073.stdout:# minimal ceph.conf for f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:19:11.015 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-03-11T22:19:11.015 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:19:11.015 INFO:teuthology.orchestra.run.smithi073.stdout: mon_host = [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2024-03-11T22:19:11.549 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:11 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2904042689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:11.549 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:11 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:11.657 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:19:11.657 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:19:11.657 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:19:11.687 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:19:11.687 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:19:11.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:11 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2904042689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:11.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:11 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:11.754 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:19:11.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:19:11.785 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:19:11.786 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:19:11.853 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:19:11.853 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:19:11.853 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:19:11.869 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:19:11.869 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-11 22:18:31.134256203 +0000 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-11 22:15:39.912714791 +0000 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-11 22:15:39.912714791 +0000 2024-03-11T22:19:11.923 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-11 22:15:39.912714791 +0000 2024-03-11T22:19:11.924 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:19:11.988 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-11T22:19:11.988 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-11T22:19:11.988 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000215295 s, 2.4 MB/s 2024-03-11T22:19:11.989 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:19:12.045 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-11 22:18:31.134256203 +0000 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-11 22:15:40.249706852 +0000 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-11 22:15:40.249706852 +0000 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-11 22:15:40.249706852 +0000 2024-03-11T22:19:12.100 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000173073 s, 3.0 MB/s 2024-03-11T22:19:12.166 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:19:12.221 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-11 22:18:31.134256203 +0000 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-11 22:15:40.597698654 +0000 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-11 22:15:40.597698654 +0000 2024-03-11T22:19:12.277 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-11 22:15:40.597698654 +0000 2024-03-11T22:19:12.277 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:19:12.342 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-11T22:19:12.342 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-11T22:19:12.342 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000219477 s, 2.3 MB/s 2024-03-11T22:19:12.343 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:19:12.399 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-03-11T22:19:12.454 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:19:12.454 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-11 22:18:31.135256179 +0000 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-11 22:15:40.951690314 +0000 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-11 22:15:40.951690314 +0000 2024-03-11T22:19:12.455 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-11 22:15:40.951690314 +0000 2024-03-11T22:19:12.455 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:19:12.520 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-11T22:19:12.520 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-11T22:19:12.520 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000189396 s, 2.7 MB/s 2024-03-11T22:19:12.521 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:19:12.577 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-11T22:19:12.577 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:19:12.593 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:19:12.593 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-03-11T22:19:12.609 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:12 smithi099 ceph-mon[27157]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-11 22:19:07.617467405 +0000 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-11 22:15:39.926273370 +0000 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-11 22:15:39.926273370 +0000 2024-03-11T22:19:12.624 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-11 22:15:39.926273370 +0000 2024-03-11T22:19:12.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:19:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-11T22:19:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-11T22:19:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000272878 s, 1.9 MB/s 2024-03-11T22:19:12.692 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:19:12.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:12 smithi073 ceph-mon[24794]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:12.747 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-03-11T22:19:12.802 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:19:12.802 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-11 22:19:07.617467405 +0000 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-11 22:15:40.282265283 +0000 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-11 22:15:40.282265283 +0000 2024-03-11T22:19:12.803 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-11 22:15:40.282265283 +0000 2024-03-11T22:19:12.803 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:19:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-11T22:19:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-11T22:19:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00022876 s, 2.2 MB/s 2024-03-11T22:19:12.869 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:19:12.926 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-03-11T22:19:12.982 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:19:12.982 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:12.982 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T22:19:12.982 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:12.983 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:12.983 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-11 22:19:07.617467405 +0000 2024-03-11T22:19:12.983 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-11 22:15:40.633257309 +0000 2024-03-11T22:19:12.983 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-11 22:15:40.633257309 +0000 2024-03-11T22:19:12.983 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-11 22:15:40.633257309 +0000 2024-03-11T22:19:12.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:19:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-11T22:19:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-11T22:19:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000218536 s, 2.3 MB/s 2024-03-11T22:19:13.050 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:19:13.106 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-03-11T22:19:13.161 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:19:13.161 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:19:13.161 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-11 22:19:07.618467382 +0000 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-11 22:15:40.982249381 +0000 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-11 22:15:40.982249381 +0000 2024-03-11T22:19:13.162 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-11 22:15:40.982249381 +0000 2024-03-11T22:19:13.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:19:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-11T22:19:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-11T22:19:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000183398 s, 2.8 MB/s 2024-03-11T22:19:13.228 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:19:13.286 INFO:tasks.cephadm:Deploying osd.0 on smithi073 with /dev/vg_nvme/lv_4... 2024-03-11T22:19:13.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:19:14.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:13 smithi073 ceph-mon[24794]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:14.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:14.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:13 smithi073 ceph-mon[24794]: Deploying daemon prometheus.smithi073 on smithi073 2024-03-11T22:19:14.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:13 smithi099 ceph-mon[27157]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:14.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:14.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:13 smithi099 ceph-mon[27157]: Deploying daemon prometheus.smithi073 on smithi073 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114999 s, 912 MB/s 2024-03-11T22:19:15.509 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-11T22:19:16.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:16 smithi073 ceph-mon[24794]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:16.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:16 smithi099 ceph-mon[27157]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:16.655 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_4 2024-03-11T22:19:17.341 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:17 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:17.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:17 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:18.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:18 smithi073 ceph-mon[24794]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:18.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:18 smithi099 ceph-mon[27157]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:19.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:19 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:19:19.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:19 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:19:19.556 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:19 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:19.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:19 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:19:19.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:19 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:19:19.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:19 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:20.620 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:20 smithi073 ceph-mon[24794]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:20.620 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:20 smithi073 ceph-mon[24794]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:20.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:20 smithi099 ceph-mon[27157]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:20.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:20 smithi099 ceph-mon[27157]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:19:22.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:22 smithi073 ceph-mon[24794]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:22.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:22 smithi099 ceph-mon[27157]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:24.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:24 smithi073 ceph-mon[24794]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:24.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:24 smithi099 ceph-mon[27157]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:26.605 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:26 smithi073 ceph-mon[24794]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:26.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:26 smithi099 ceph-mon[27157]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:28.704 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:28 smithi073 ceph-mon[24794]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:28.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:28 smithi099 ceph-mon[27157]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:30.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:30 smithi099 ceph-mon[27157]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:30.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:30.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:30.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:30.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:30.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:30 smithi073 ceph-mon[24794]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:30.570 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:30.571 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:30.571 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:30.571 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3956998706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ade19a11-36e5-4623-bd98-f6e83bd1ae95"}]: dispatch 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3956998706' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ade19a11-36e5-4623-bd98-f6e83bd1ae95"}]': finished 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:32.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:32.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:19:32.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:32 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3624635691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:19:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3956998706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ade19a11-36e5-4623-bd98-f6e83bd1ae95"}]: dispatch 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3956998706' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ade19a11-36e5-4623-bd98-f6e83bd1ae95"}]': finished 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:19:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:32 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3624635691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:19:33.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:33 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:19:33.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:33.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:33 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:19:33.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:34.536 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:34 smithi073 ceph-mon[24794]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:34.536 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:34 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:34.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:34 smithi099 ceph-mon[27157]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:34.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:34 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: Reconfiguring mon.smithi073 (unknown last config time)... 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:35.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:35 smithi073 ceph-mon[24794]: Reconfiguring daemon mon.smithi073 on smithi073 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: Reconfiguring mon.smithi073 (unknown last config time)... 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:35.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:35 smithi099 ceph-mon[27157]: Reconfiguring daemon mon.smithi073 on smithi073 2024-03-11T22:19:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:36 smithi099 ceph-mon[27157]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:19:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:19:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:36.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:36 smithi073 ceph-mon[24794]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:36.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:36.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:19:36.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:19:36.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:37.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:37 smithi073 ceph-mon[24794]: Reconfiguring mgr.smithi073.pglkvm (unknown last config time)... 2024-03-11T22:19:37.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:37 smithi073 ceph-mon[24794]: Reconfiguring daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:19:37.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:37 smithi099 ceph-mon[27157]: Reconfiguring mgr.smithi073.pglkvm (unknown last config time)... 2024-03-11T22:19:37.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:37 smithi099 ceph-mon[27157]: Reconfiguring daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:19:38.623 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:38 smithi073 ceph-mon[24794]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:38.624 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:38 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:38 smithi099 ceph-mon[27157]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:38 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:39.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:39 smithi073 ceph-mon[24794]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-03-11T22:19:39.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:39 smithi073 ceph-mon[24794]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-03-11T22:19:39.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:39 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:19:39.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:39 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:39.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:39 smithi099 ceph-mon[27157]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-03-11T22:19:39.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:39 smithi099 ceph-mon[27157]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-03-11T22:19:39.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:39 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:19:39.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:39 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:40.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:40 smithi099 ceph-mon[27157]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:40.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:40 smithi099 ceph-mon[27157]: Deploying daemon osd.0 on smithi073 2024-03-11T22:19:40.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:40 smithi073 ceph-mon[24794]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:40.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:40 smithi073 ceph-mon[24794]: Deploying daemon osd.0 on smithi073 2024-03-11T22:19:41.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:19:41.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:19:41.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:19:41.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:19:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:42 smithi099 ceph-mon[27157]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:42.805 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:42 smithi073 ceph-mon[24794]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:44.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:44 smithi073 ceph-mon[24794]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:44.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:44 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:44.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:44 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:44.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:44 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:44.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:44 smithi099 ceph-mon[27157]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:44.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:44 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:44.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:44 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:44.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:44 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:45.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:45 smithi073 ceph-mon[24794]: Reconfiguring crash.smithi073 (monmap changed)... 2024-03-11T22:19:45.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:45 smithi073 ceph-mon[24794]: Reconfiguring daemon crash.smithi073 on smithi073 2024-03-11T22:19:45.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:45 smithi099 ceph-mon[27157]: Reconfiguring crash.smithi073 (monmap changed)... 2024-03-11T22:19:45.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:45 smithi099 ceph-mon[27157]: Reconfiguring daemon crash.smithi073 on smithi073 2024-03-11T22:19:46.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:46 smithi073 ceph-mon[24794]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:46.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:46 smithi099 ceph-mon[27157]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:48.768 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:48 smithi073 ceph-mon[24794]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:48.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:48 smithi099 ceph-mon[27157]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:50.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:49 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:50.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:49 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:50.956 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:50 smithi073 ceph-mon[24794]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:50.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:50 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:51.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:50 smithi099 ceph-mon[27157]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:51.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:50 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:51.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:51 smithi073 ceph-mon[24794]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-03-11T22:19:51.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:51 smithi073 ceph-mon[24794]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-03-11T22:19:52.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:51 smithi099 ceph-mon[27157]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-03-11T22:19:52.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:51 smithi099 ceph-mon[27157]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-03-11T22:19:53.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:52 smithi073 ceph-mon[24794]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:53.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:52 smithi073 ceph-mon[24794]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:19:53.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:52 smithi099 ceph-mon[27157]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:53.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:52 smithi099 ceph-mon[27157]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:19:54.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:53 smithi073 ceph-mon[24794]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:19:54.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:53 smithi073 ceph-mon[24794]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:19:54.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:53 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:54.101 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:53 smithi073 ceph-mon[24794]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:19:54.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:53 smithi099 ceph-mon[27157]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:19:54.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:53 smithi099 ceph-mon[27157]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:19:54.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:53 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:54.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:53 smithi099 ceph-mon[27157]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:19:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:54.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: osdmap e8: 1 total, 0 up, 1 in 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:54.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:55.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: osdmap e8: 1 total, 0 up, 1 in 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:19:55.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:55.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:19:55.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: Reconfiguring crash.smithi099 (monmap changed)... 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: Reconfiguring daemon crash.smithi099 on smithi099 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743] boot 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:19:55.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: Reconfiguring crash.smithi099 (monmap changed)... 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: Reconfiguring daemon crash.smithi099 on smithi099 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: osd.0 [v2:172.21.15.73:6802/296531743,v1:172.21.15.73:6803/296531743] boot 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:19:56.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:19:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:19:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:19:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:57.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:56 smithi099 ceph-mon[27157]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:57.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:56 smithi099 ceph-mon[27157]: Reconfiguring mgr.smithi099.rcoixq (monmap changed)... 2024-03-11T22:19:57.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:56 smithi099 ceph-mon[27157]: Reconfiguring daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:19:57.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:56 smithi099 ceph-mon[27157]: osdmap e10: 1 total, 1 up, 1 in 2024-03-11T22:19:57.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:56 smithi073 ceph-mon[24794]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:19:57.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:56 smithi073 ceph-mon[24794]: Reconfiguring mgr.smithi099.rcoixq (monmap changed)... 2024-03-11T22:19:57.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:56 smithi073 ceph-mon[24794]: Reconfiguring daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:19:57.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:56 smithi073 ceph-mon[24794]: osdmap e10: 1 total, 1 up, 1 in 2024-03-11T22:19:57.303 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 0 on host 'smithi073' 2024-03-11T22:19:57.883 DEBUG:teuthology.orchestra.run.smithi073:osd.0> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:19:57.887 INFO:tasks.cephadm:Deploying osd.1 on smithi073 with /dev/vg_nvme/lv_3... 2024-03-11T22:19:57.887 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: Reconfiguring mon.smithi099 (monmap changed)... 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: Reconfiguring daemon mon.smithi099 on smithi099 2024-03-11T22:19:58.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: Reconfiguring mon.smithi099 (monmap changed)... 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: Reconfiguring daemon mon.smithi099 on smithi099 2024-03-11T22:19:58.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.73:9093"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.73:9093"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.73:3000"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.73:3000"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:19:59.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.73:9095"}]: dispatch 2024-03-11T22:19:59.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.73:9095"}]: dispatch 2024-03-11T22:19:59.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:59.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:59.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:19:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:59.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.73:9093"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.73:9093"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.73:3000"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.73:3000"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.73:9095"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.73:9095"}]: dispatch 2024-03-11T22:19:59.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:19:59.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:59.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:59.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:19:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:59.914 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-11T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-11T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134579 s, 779 MB/s 2024-03-11T22:19:59.915 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-11T22:20:00.589 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:00 smithi073 ceph-mon[24794]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:00.589 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:00 smithi073 ceph-mon[24794]: overall HEALTH_OK 2024-03-11T22:20:00.589 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:00.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:00 smithi099 ceph-mon[27157]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:00.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:00 smithi099 ceph-mon[27157]: overall HEALTH_OK 2024-03-11T22:20:00.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:00.970 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_3 2024-03-11T22:20:02.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:02 smithi099 ceph-mon[27157]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:02.749 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:02 smithi073 ceph-mon[24794]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:04.759 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:04 smithi073 ceph-mon[24794]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:04.759 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:04 smithi073 ceph-mon[24794]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:04.759 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:04.759 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:04.759 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:04.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:04 smithi099 ceph-mon[27157]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:04.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:04 smithi099 ceph-mon[27157]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:04.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:04.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:04.802 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:06.439 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:06 smithi073 ceph-mon[24794]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:06.439 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:06 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:06.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:06 smithi099 ceph-mon[27157]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:06.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:06 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:07.378 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:07 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1860165451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ead05ba-132c-4a2d-b442-95a6ed3ce29a"}]: dispatch 2024-03-11T22:20:07.379 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:07 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1860165451' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ead05ba-132c-4a2d-b442-95a6ed3ce29a"}]': finished 2024-03-11T22:20:07.379 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:07 smithi073 ceph-mon[24794]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:20:07.379 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:07.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:07 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1860165451' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ead05ba-132c-4a2d-b442-95a6ed3ce29a"}]: dispatch 2024-03-11T22:20:07.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:07 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1860165451' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ead05ba-132c-4a2d-b442-95a6ed3ce29a"}]': finished 2024-03-11T22:20:07.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:07 smithi099 ceph-mon[27157]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:20:07.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:08.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:08 smithi073 ceph-mon[24794]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:08.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:08 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/4127417945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:20:08.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:08 smithi099 ceph-mon[27157]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:08.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:08 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/4127417945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:20:10.456 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:10 smithi073 ceph-mon[24794]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:10.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:10 smithi099 ceph-mon[27157]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:12.572 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:12 smithi073 ceph-mon[24794]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:12.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:12 smithi099 ceph-mon[27157]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi073 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi073 to 12864M 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:13.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:13.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi073 2024-03-11T22:20:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi073 to 12864M 2024-03-11T22:20:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:14.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:14 smithi073 ceph-mon[24794]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:14.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:14.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:14 smithi099 ceph-mon[27157]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:14.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:15.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:15 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:20:15.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:15 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:15.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:15 smithi073 ceph-mon[24794]: Deploying daemon osd.1 on smithi073 2024-03-11T22:20:15.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:15 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:20:15.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:15 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:15.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:15 smithi099 ceph-mon[27157]: Deploying daemon osd.1 on smithi073 2024-03-11T22:20:16.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:16 smithi073 ceph-mon[24794]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:16.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:16 smithi099 ceph-mon[27157]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:18.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:18 smithi073 ceph-mon[24794]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:18.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:18 smithi099 ceph-mon[27157]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:20.599 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:20 smithi073 ceph-mon[24794]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:20.599 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:20 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:20.599 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:20 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:20.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:20 smithi099 ceph-mon[27157]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:20.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:20 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:20.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:20 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:22.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:22 smithi099 ceph-mon[27157]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:22.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:22.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:22.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:22.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:22.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:22 smithi073 ceph-mon[24794]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:22.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:22.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:22.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:22.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:23.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:23 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:24.022 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 1 on host 'smithi073' 2024-03-11T22:20:24.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:23 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:24.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:24 smithi073 ceph-mon[24794]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:24.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:24.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:24 smithi073 ceph-mon[24794]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:20:24.842 DEBUG:teuthology.orchestra.run.smithi073:osd.1> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:20:24.845 INFO:tasks.cephadm:Deploying osd.2 on smithi073 with /dev/vg_nvme/lv_2... 2024-03-11T22:20:24.845 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:20:25.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:24 smithi099 ceph-mon[27157]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:25.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:25.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:24 smithi099 ceph-mon[27157]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:20:26.286 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:26 smithi099 ceph-mon[27157]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:26.286 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:26 smithi099 ceph-mon[27157]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:20:26.287 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:26 smithi099 ceph-mon[27157]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:20:26.287 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:26 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:26.287 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:26 smithi099 ceph-mon[27157]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:20:26.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-mon[24794]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:26.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-mon[24794]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:20:26.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-mon[24794]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:20:26.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:26.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-mon[24794]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:20:26.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[39266]: 2024-03-11T22:20:26.026+0000 7fe0fc6bf700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:20:26.315 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:26 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[39266]: 2024-03-11T22:20:26.030+0000 7fe0f7056700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:20:27.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:27 smithi073 ceph-mon[24794]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:20:27.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:27 smithi073 ceph-mon[24794]: osdmap e13: 2 total, 1 up, 2 in 2024-03-11T22:20:27.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:27.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:27.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:27 smithi099 ceph-mon[27157]: from='osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:20:27.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:27 smithi099 ceph-mon[27157]: osdmap e13: 2 total, 1 up, 2 in 2024-03-11T22:20:27.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:27.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:27.926 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:20:27.927 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:20:27.927 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-11T22:20:27.927 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-11T22:20:27.927 INFO:teuthology.orchestra.run.smithi073.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128612 s, 815 MB/s 2024-03-11T22:20:27.927 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-11T22:20:28.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:20:28.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:20:28.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:28.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:28.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266] boot 2024-03-11T22:20:28.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:20:28.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:28.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:20:28.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:20:28.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:20:28.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:28.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: osd.1 [v2:172.21.15.73:6810/1931878266,v1:172.21.15.73:6811/1931878266] boot 2024-03-11T22:20:28.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:20:28.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:20:28.669 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_2 2024-03-11T22:20:29.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:29 smithi073 ceph-mon[24794]: osdmap e15: 2 total, 2 up, 2 in 2024-03-11T22:20:29.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:29 smithi099 ceph-mon[27157]: osdmap e15: 2 total, 2 up, 2 in 2024-03-11T22:20:30.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:30 smithi073 ceph-mon[24794]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:30.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:30 smithi099 ceph-mon[27157]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:32.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:32.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:32.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:32.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:32.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:32.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:32.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:32.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:34.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:34 smithi073 ceph-mon[24794]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:34.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:34 smithi099 ceph-mon[27157]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:36.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi073 to 6432M 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:36.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:36.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi073 to 6432M 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:36.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:37.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1346024068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef61f19-bf1e-4f37-b2dc-c33ca51b85ae"}]: dispatch 2024-03-11T22:20:37.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1346024068' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cef61f19-bf1e-4f37-b2dc-c33ca51b85ae"}]': finished 2024-03-11T22:20:37.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:37 smithi073 ceph-mon[24794]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:20:37.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:37 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:37.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2341191621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:20:37.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:37 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1346024068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef61f19-bf1e-4f37-b2dc-c33ca51b85ae"}]: dispatch 2024-03-11T22:20:37.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:37 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1346024068' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cef61f19-bf1e-4f37-b2dc-c33ca51b85ae"}]': finished 2024-03-11T22:20:37.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:37 smithi099 ceph-mon[27157]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:20:37.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:37 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:37.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:37 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2341191621' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:20:38.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:38 smithi073 ceph-mon[24794]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:38.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:38 smithi099 ceph-mon[27157]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:41.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:40 smithi073 ceph-mon[24794]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:41.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:41.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:41.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:40 smithi099 ceph-mon[27157]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:41.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:41.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:41.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:20:41.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:20:42.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:20:42.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:20:42.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:42 smithi073 ceph-mon[24794]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:42.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:20:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:43.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:42 smithi099 ceph-mon[27157]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:43.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:20:43.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:43.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:43 smithi073 ceph-mon[24794]: Deploying daemon osd.2 on smithi073 2024-03-11T22:20:44.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:43 smithi099 ceph-mon[27157]: Deploying daemon osd.2 on smithi073 2024-03-11T22:20:44.952 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:44 smithi073 ceph-mon[24794]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:45.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:44 smithi099 ceph-mon[27157]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:46.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:46 smithi073 ceph-mon[24794]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:47.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:46 smithi099 ceph-mon[27157]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:48.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:48 smithi073 ceph-mon[24794]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:48.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:48 smithi099 ceph-mon[27157]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:48.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:48 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:48.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:48 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:48 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:48 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:50.546 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:50 smithi073 ceph-mon[24794]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:50.575 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 2 on host 'smithi073' 2024-03-11T22:20:50.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:50 smithi099 ceph-mon[27157]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:51.215 DEBUG:teuthology.orchestra.run.smithi073:osd.2> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:20:51.217 INFO:tasks.cephadm:Deploying osd.3 on smithi073 with /dev/vg_nvme/lv_1... 2024-03-11T22:20:51.217 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T22:20:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:51 smithi073 ceph-mon[24794]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:20:51.464 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:51.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:51 smithi099 ceph-mon[27157]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:20:51.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:51 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-mon[24794]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-mon[24794]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:20:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-mon[24794]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:20:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-mon[24794]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:20:52.715 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[42982]: 2024-03-11T22:20:52.359+0000 7fec0bc24700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:20:52.715 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:52 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[42982]: 2024-03-11T22:20:52.362+0000 7fec07dbe700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:20:52.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:52 smithi099 ceph-mon[27157]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:52.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:52 smithi099 ceph-mon[27157]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:20:52.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:52 smithi099 ceph-mon[27157]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:20:52.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:52 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:52.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:52 smithi099 ceph-mon[27157]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:20:53.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:53 smithi073 ceph-mon[24794]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:20:53.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:53 smithi073 ceph-mon[24794]: osdmap e18: 3 total, 2 up, 3 in 2024-03-11T22:20:53.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:53 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:53.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:53 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:53.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:53 smithi099 ceph-mon[27157]: from='osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:20:53.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:53 smithi099 ceph-mon[27157]: osdmap e18: 3 total, 2 up, 3 in 2024-03-11T22:20:53.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:53 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:53.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:53 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128197 s, 818 MB/s 2024-03-11T22:20:53.769 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795] boot 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:20:54.403 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:54.686 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_1 2024-03-11T22:20:54.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:20:54.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:20:54.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:20:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: osd.2 [v2:172.21.15.73:6818/2023932795,v1:172.21.15.73:6819/2023932795] boot 2024-03-11T22:20:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:20:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:20:55.635 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:55 smithi073 ceph-mon[24794]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:20:55.636 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T22:20:55.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:55 smithi099 ceph-mon[27157]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:20:55.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T22:20:56.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:56 smithi073 ceph-mon[24794]: pgmap v75: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:56.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:56 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T22:20:56.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:56 smithi073 ceph-mon[24794]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:20:56.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:56 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:20:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:56 smithi099 ceph-mon[27157]: pgmap v75: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:56 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T22:20:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:56 smithi099 ceph-mon[27157]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:20:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:56 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:20:57.364 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44409]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:20:57.364 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44409]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:57.364 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44409]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:57.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44420]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:20:57.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44420]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:57.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44420]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:57.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:57 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:20:57.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:57 smithi073 ceph-mon[24794]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:20:57.700 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44508]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:20:57.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:57 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:20:57.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:57 smithi099 ceph-mon[27157]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:20:57.964 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44508]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:57.965 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44508]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:58.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44588]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:20:58.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44588]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:58.230 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44588]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:58.230 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44559]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:20:58.230 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44559]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:58.231 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:20:57 smithi073 sudo[44559]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:58.481 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:58.731 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44608]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:20:58.731 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44608]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:58.732 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44608]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:58.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:20:58.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:20:58.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:58.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:20:58.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:20:58.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:20:58.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:20:58.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:58 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:58.994 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44630]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:20:58.994 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44630]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:58.995 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44630]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:59.334 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44638]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:20:59.335 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44638]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:59.335 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:58 smithi073 sudo[44638]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:59.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29486]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:20:59.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29486]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:59.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29486]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: mgrmap e19: smithi073.pglkvm(active, since 2m), standbys: smithi099.rcoixq 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:20:59.630 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:20:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:20:59.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29489]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29489]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 sudo[29489]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: mgrmap e19: smithi073.pglkvm(active, since 2m), standbys: smithi099.rcoixq 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:20:59.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:20:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:21:00.584 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:00.584 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:21:00.584 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:21:00.584 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:00.585 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:00 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:00.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:00.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:00.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:00 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:01.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:01 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi073 2024-03-11T22:21:01.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:01 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi073 to 4288M 2024-03-11T22:21:01.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:01 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi073 2024-03-11T22:21:01.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:01 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi073 to 4288M 2024-03-11T22:21:02.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1095862008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa5a768-6564-4c73-9414-06634f937eac"}]: dispatch 2024-03-11T22:21:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1095862008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa5a768-6564-4c73-9414-06634f937eac"}]': finished 2024-03-11T22:21:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:21:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:02 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3623779304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:02.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:02.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1095862008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa5a768-6564-4c73-9414-06634f937eac"}]: dispatch 2024-03-11T22:21:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1095862008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa5a768-6564-4c73-9414-06634f937eac"}]': finished 2024-03-11T22:21:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:21:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:02 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3623779304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:04.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:04 smithi073 ceph-mon[24794]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:04.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:04 smithi099 ceph-mon[27157]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:06.628 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:06 smithi073 ceph-mon[24794]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:21:06.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:06 smithi099 ceph-mon[27157]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:21:08.705 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:08 smithi073 ceph-mon[24794]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T22:21:08.705 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:21:08.705 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:08.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:08 smithi099 ceph-mon[27157]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T22:21:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:21:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:09.615 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:09 smithi073 ceph-mon[24794]: Deploying daemon osd.3 on smithi073 2024-03-11T22:21:09.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:09 smithi099 ceph-mon[27157]: Deploying daemon osd.3 on smithi073 2024-03-11T22:21:10.694 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:10 smithi073 ceph-mon[24794]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:21:10.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:10 smithi099 ceph-mon[27157]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:21:12.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:12 smithi099 ceph-mon[27157]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:21:12.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:12 smithi073 ceph-mon[24794]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:21:14.350 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:14 smithi073 ceph-mon[24794]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-03-11T22:21:14.350 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:14.351 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:14.351 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:14.351 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:14.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:14 smithi099 ceph-mon[27157]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 4 op/s 2024-03-11T22:21:14.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:14.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:14.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:14.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:16.246 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:16 smithi073 ceph-mon[24794]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T22:21:16.278 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 3 on host 'smithi073' 2024-03-11T22:21:16.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:16 smithi099 ceph-mon[27157]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T22:21:16.959 DEBUG:teuthology.orchestra.run.smithi073:osd.3> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T22:21:16.961 INFO:tasks.cephadm:Deploying osd.4 on smithi099 with /dev/vg_nvme/lv_4... 2024-03-11T22:21:16.961 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:21:17.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:17 smithi073 ceph-mon[24794]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:21:17.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:17 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:17.271 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:17 smithi099 ceph-mon[27157]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:21:17.272 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:17 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:18.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-mon[24794]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:18.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-mon[24794]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:21:18.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-mon[24794]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:21:18.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:18.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-mon[24794]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:21:18.215 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[46258]: 2024-03-11T22:21:18.055+0000 7fa2f80e2700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T22:21:18.215 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:21:18 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[46258]: 2024-03-11T22:21:18.061+0000 7fa2f1276700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:21:18.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:18 smithi099 ceph-mon[27157]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:18.312 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:18 smithi099 ceph-mon[27157]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:21:18.312 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:18 smithi099 ceph-mon[27157]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:21:18.312 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:18 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:18.312 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:18 smithi099 ceph-mon[27157]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:21:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:21:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:21:18.889 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-03-11T22:21:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-03-11T22:21:18.889 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114147 s, 919 MB/s 2024-03-11T22:21:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-03-11T22:21:19.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:19 smithi099 ceph-mon[27157]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:21:19.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:19 smithi099 ceph-mon[27157]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T22:21:19.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:19 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:19.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:19 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:19.415 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:19 smithi073 ceph-mon[24794]: from='osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-11T22:21:19.415 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:19 smithi073 ceph-mon[24794]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T22:21:19.415 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:19 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:19.415 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:19 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:19.458 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_4 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667] boot 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:20.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:20 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: osd.3 [v2:172.21.15.73:6826/1137804667,v1:172.21.15.73:6827/1137804667] boot 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:21:20.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:20 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:21.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:21 smithi073 ceph-mon[24794]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T22:21:21.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:21 smithi099 ceph-mon[27157]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T22:21:22.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:22 smithi099 ceph-mon[27157]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:22.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:22 smithi099 ceph-mon[27157]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:21:22.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:21:22.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:21:22.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:22.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:22 smithi073 ceph-mon[24794]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:22.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:22 smithi073 ceph-mon[24794]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:21:22.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:21:22.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:21:22.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi073 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi073 to 3216M 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3241318740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]': finished 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:24 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3645515120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi073 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi073 to 3216M 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3241318740' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d27896cd-413d-4fb3-832e-5b1c25b95dec"}]': finished 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:24 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3645515120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:26 smithi073 ceph-mon[24794]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:26.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:26 smithi099 ceph-mon[27157]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:28 smithi099 ceph-mon[27157]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:28.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:28 smithi073 ceph-mon[24794]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:30 smithi073 ceph-mon[24794]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:21:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:30 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:30 smithi073 ceph-mon[24794]: Deploying daemon osd.4 on smithi099 2024-03-11T22:21:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:30 smithi099 ceph-mon[27157]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:21:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:30 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:30 smithi099 ceph-mon[27157]: Deploying daemon osd.4 on smithi099 2024-03-11T22:21:32.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:32 smithi099 ceph-mon[27157]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:32.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:32 smithi073 ceph-mon[24794]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:34.688 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:34 smithi099 ceph-mon[27157]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:34.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:34 smithi073 ceph-mon[24794]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:35.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:35.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:35.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:35.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:35.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:35.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:35.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:35.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:36.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:36 smithi073 ceph-mon[24794]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:36.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:36 smithi099 ceph-mon[27157]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:37.637 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 4 on host 'smithi099' 2024-03-11T22:21:38.266 DEBUG:teuthology.orchestra.run.smithi099:osd.4> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T22:21:38.268 INFO:tasks.cephadm:Deploying osd.5 on smithi099 with /dev/vg_nvme/lv_3... 2024-03-11T22:21:38.268 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:21:38.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:38 smithi073 ceph-mon[24794]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:38.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:38 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:38.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:38 smithi073 ceph-mon[24794]: from='osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:21:38.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:38 smithi073 ceph-mon[24794]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:21:38.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:38 smithi099 ceph-mon[27157]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:38.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:38 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:38.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:38 smithi099 ceph-mon[27157]: from='osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:21:38.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:38 smithi099 ceph-mon[27157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:21:39.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:39 smithi073 ceph-mon[24794]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:21:39.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:39 smithi073 ceph-mon[24794]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T22:21:39.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:39 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:39.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:39 smithi073 ceph-mon[24794]: from='osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:21:39.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:39 smithi073 ceph-mon[24794]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:21:39.978 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-mon[27157]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:21:39.978 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-mon[27157]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T22:21:39.978 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:39.979 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-mon[27157]: from='osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:21:39.979 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-mon[27157]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:21:39.979 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[31406]: 2024-03-11T22:21:39.659+0000 7fbb7c005700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T22:21:39.979 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:21:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[31406]: 2024-03-11T22:21:39.664+0000 7fbb789a0700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:21:40.779 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:21:40.780 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:21:40.780 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-03-11T22:21:40.780 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-03-11T22:21:40.780 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127511 s, 822 MB/s 2024-03-11T22:21:40.780 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:40.968 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:41.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:21:41.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:21:41.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863] boot 2024-03-11T22:21:41.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T22:21:41.929 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:41.929 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:21:41.929 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: osd.4 [v2:172.21.15.99:6800/2864491863,v1:172.21.15.99:6801/2864491863] boot 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:21:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:21:42.710 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_3 2024-03-11T22:21:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:42 smithi073 ceph-mon[24794]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:42 smithi073 ceph-mon[24794]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T22:21:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:42 smithi073 ceph-mon[24794]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T22:21:42.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:42 smithi099 ceph-mon[27157]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-11T22:21:42.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:42 smithi099 ceph-mon[27157]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T22:21:42.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:42 smithi099 ceph-mon[27157]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T22:21:44.671 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:44 smithi099 ceph-mon[27157]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:44.671 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:44 smithi099 ceph-mon[27157]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:21:44.714 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:44 smithi073 ceph-mon[24794]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:44.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:44 smithi073 ceph-mon[24794]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi099 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi099 to 15936M 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:45.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:45.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:45.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:45 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:21:45.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi099 2024-03-11T22:21:45.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:21:45.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi099 to 15936M 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:45.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:45 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:46.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:46 smithi099 ceph-mon[27157]: pgmap v116: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:21:46.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:46 smithi099 ceph-mon[27157]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:21:46.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:46 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:21:46.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:46 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:21:46.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:46 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:46.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:46 smithi073 ceph-mon[24794]: pgmap v116: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:21:46.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:46 smithi073 ceph-mon[24794]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:21:46.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:46 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:21:46.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:46 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:21:46.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:46 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/1459478333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]: dispatch 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]: dispatch 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]': finished 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:21:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:48 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/4090373048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/1459478333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]: dispatch 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]: dispatch 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21863417-f4e5-4f0e-b42c-edcf00f14a0e"}]': finished 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T22:21:48.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:21:48.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:48 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/4090373048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:21:50.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:50 smithi073 ceph-mon[24794]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T22:21:50.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:50 smithi099 ceph-mon[27157]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T22:21:52.689 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:52 smithi099 ceph-mon[27157]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:52.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:52 smithi073 ceph-mon[24794]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:54.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:54 smithi073 ceph-mon[24794]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:54.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:21:54.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:54.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:54 smithi073 ceph-mon[24794]: Deploying daemon osd.5 on smithi099 2024-03-11T22:21:54.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:54 smithi099 ceph-mon[27157]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:21:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:54 smithi099 ceph-mon[27157]: Deploying daemon osd.5 on smithi099 2024-03-11T22:21:56.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:56 smithi099 ceph-mon[27157]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:56.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:56 smithi073 ceph-mon[24794]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:58.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:58 smithi099 ceph-mon[27157]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:58.616 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:58 smithi073 ceph-mon[24794]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:21:59.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:21:59.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:59.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:59.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:21:59 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:21:59 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:01.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:00 smithi073 ceph-mon[24794]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:01.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:00 smithi099 ceph-mon[27157]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:01.563 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 5 on host 'smithi099' 2024-03-11T22:22:02.476 DEBUG:teuthology.orchestra.run.smithi099:osd.5> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T22:22:02.478 INFO:tasks.cephadm:Deploying osd.6 on smithi099 with /dev/vg_nvme/lv_2... 2024-03-11T22:22:02.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:22:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:02 smithi099 ceph-mon[27157]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:02 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:02 smithi099 ceph-mon[27157]: from='osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:22:02.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:02 smithi099 ceph-mon[27157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:22:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:02 smithi073 ceph-mon[24794]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:02 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:02 smithi073 ceph-mon[24794]: from='osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:22:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:02 smithi073 ceph-mon[24794]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:22:03.937 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:22:03.937 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:22:03.937 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:03.937 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: from='osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:03.937 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:03.938 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:03.938 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[34368]: 2024-03-11T22:22:03.768+0000 7f805b8d9700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T22:22:03.938 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:22:03 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[34368]: 2024-03-11T22:22:03.773+0000 7f805426c700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:22:04.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:22:04.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:22:04.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:04.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: from='osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:04.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:04.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:03 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:04.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:04 smithi099 ceph-mon[27157]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:04.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:04 smithi099 ceph-mon[27157]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:04.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:04 smithi099 ceph-mon[27157]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:22:04.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:04.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:05.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:04 smithi073 ceph-mon[24794]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:22:05.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:04 smithi073 ceph-mon[24794]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:05.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:04 smithi073 ceph-mon[24794]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:22:05.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:05.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:22:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:22:05.931 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-03-11T22:22:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-03-11T22:22:05.932 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126534 s, 829 MB/s 2024-03-11T22:22:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098] boot 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:06.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:05 smithi099 ceph-mon[27157]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:22:06.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:22:06.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:22:06.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: osd.5 [v2:172.21.15.99:6808/390666098,v1:172.21.15.99:6809/390666098] boot 2024-03-11T22:22:06.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T22:22:06.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:22:06.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:05 smithi073 ceph-mon[24794]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:22:06.729 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_2 2024-03-11T22:22:06.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:06 smithi099 ceph-mon[27157]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:06.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:06 smithi099 ceph-mon[27157]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:22:07.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:06 smithi073 ceph-mon[24794]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:07.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:06 smithi073 ceph-mon[24794]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:22:07.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:07 smithi099 ceph-mon[27157]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:22:07.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:07 smithi099 ceph-mon[27157]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T22:22:08.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:07 smithi073 ceph-mon[24794]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:22:08.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:07 smithi073 ceph-mon[24794]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T22:22:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:08.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:08.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:08 smithi099 ceph-mon[27157]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:09.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:09.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:09.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:09.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:09.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:08 smithi073 ceph-mon[24794]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T22:22:10.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:09 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi099 2024-03-11T22:22:10.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:09 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi099 to 7968M 2024-03-11T22:22:10.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:09 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:22:10.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:09 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:22:10.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:09 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:10.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:09 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi099 2024-03-11T22:22:10.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:09 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi099 to 7968M 2024-03-11T22:22:10.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:09 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:22:10.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:09 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:22:10.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:09 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:10.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:10 smithi099 ceph-mon[27157]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:10.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:10 smithi099 ceph-mon[27157]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:22:11.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:10 smithi073 ceph-mon[24794]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:11.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:10 smithi073 ceph-mon[24794]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:22:12.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:12.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:12.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3989979342' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]: dispatch 2024-03-11T22:22:12.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]: dispatch 2024-03-11T22:22:12.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]': finished 2024-03-11T22:22:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T22:22:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:22:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: Cluster is now healthy 2024-03-11T22:22:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:12 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/3283820843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:22:12.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:12.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:12.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3989979342' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]: dispatch 2024-03-11T22:22:12.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]: dispatch 2024-03-11T22:22:12.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f9d2ebd-1391-451a-a6f6-99e279862cf0"}]': finished 2024-03-11T22:22:12.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T22:22:12.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:12.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:22:12.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: Cluster is now healthy 2024-03-11T22:22:12.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:12 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/3283820843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:22:14.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:14 smithi099 ceph-mon[27157]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T22:22:14.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:14 smithi073 ceph-mon[24794]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T22:22:16.460 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:16 smithi099 ceph-mon[27157]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T22:22:16.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:16 smithi073 ceph-mon[24794]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T22:22:18.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:18 smithi099 ceph-mon[27157]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-11T22:22:18.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:18 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:22:18.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:18 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:18.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:18 smithi099 ceph-mon[27157]: Deploying daemon osd.6 on smithi099 2024-03-11T22:22:18.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:18 smithi073 ceph-mon[24794]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-11T22:22:18.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:18 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:22:18.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:18 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:18.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:18 smithi073 ceph-mon[24794]: Deploying daemon osd.6 on smithi099 2024-03-11T22:22:20.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:20 smithi099 ceph-mon[27157]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T22:22:20.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:20 smithi073 ceph-mon[24794]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-11T22:22:22.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:22 smithi099 ceph-mon[27157]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:22.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:22 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:22.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:22 smithi073 ceph-mon[24794]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:22.696 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:22 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:23.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:23 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:23.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:23 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:23.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:23 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:23.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:23 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:23.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:23 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:23.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:23 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:24.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:24 smithi099 ceph-mon[27157]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:24.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:24 smithi073 ceph-mon[24794]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:25.780 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 6 on host 'smithi099' 2024-03-11T22:22:26.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:26 smithi099 ceph-mon[27157]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:26.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:26 smithi099 ceph-mon[27157]: from='osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:22:26.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:26 smithi099 ceph-mon[27157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:22:26.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:26 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:26.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:26 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:26.398 DEBUG:teuthology.orchestra.run.smithi099:osd.6> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T22:22:26.401 INFO:tasks.cephadm:Deploying osd.7 on smithi099 with /dev/vg_nvme/lv_1... 2024-03-11T22:22:26.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T22:22:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:26 smithi073 ceph-mon[24794]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:26 smithi073 ceph-mon[24794]: from='osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:22:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:26 smithi073 ceph-mon[24794]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:22:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:26 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:26.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:26 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:27.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-mon[27157]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:22:27.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-mon[27157]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T22:22:27.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:27.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-mon[27157]: from='osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:27.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-mon[27157]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:27.489 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[37346]: 2024-03-11T22:22:27.368+0000 7fc915fff700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T22:22:27.489 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:22:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[37346]: 2024-03-11T22:22:27.374+0000 7fc90f193700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:22:27.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:27 smithi073 ceph-mon[24794]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:22:27.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:27 smithi073 ceph-mon[24794]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T22:22:27.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:27.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:27 smithi073 ceph-mon[24794]: from='osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:27.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:27 smithi073 ceph-mon[24794]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709] boot 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T22:22:28.445 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.616 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: osd.6 [v2:172.21.15.99:6816/3496847709,v1:172.21.15.99:6817/3496847709] boot 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T22:22:28.617 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:22:28.641 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T22:22:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T22:22:28.642 INFO:teuthology.orchestra.run.smithi099.stderr: stderr: 10+0 records in 2024-03-11T22:22:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:10+0 records out 2024-03-11T22:22:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123344 s, 850 MB/s 2024-03-11T22:22:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:--> Zapping successful for: 2024-03-11T22:22:29.305 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi099:vg_nvme/lv_1 2024-03-11T22:22:29.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:29 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:22:29.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:29 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:22:29.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:29 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:29.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:29 smithi099 ceph-mon[27157]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T22:22:29.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:29 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:22:29.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:29 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:22:29.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:29 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:29.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:29 smithi073 ceph-mon[24794]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T22:22:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:30 smithi073 ceph-mon[24794]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:30 smithi073 ceph-mon[24794]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T22:22:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:30 smithi099 ceph-mon[27157]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:30 smithi099 ceph-mon[27157]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T22:22:32.674 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:32 smithi099 ceph-mon[27157]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:22:32.674 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:32 smithi099 ceph-mon[27157]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T22:22:32.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:32 smithi073 ceph-mon[24794]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:22:32.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:32 smithi073 ceph-mon[24794]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:33.599 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:33 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T22:22:33.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:22:33.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:33.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:33 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:34.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:34 smithi099 ceph-mon[27157]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:22:34.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:34 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi099 2024-03-11T22:22:34.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:34 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi099 to 5312M 2024-03-11T22:22:34.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:34 smithi073 ceph-mon[24794]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-11T22:22:34.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:34 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi099 2024-03-11T22:22:34.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:34 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi099 to 5312M 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/2868712739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]: dispatch 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]: dispatch 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]': finished 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:36 smithi099 ceph-mon[27157]: from='client.? 172.21.15.99:0/877359255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:22:36.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T22:22:36.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/2868712739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]: dispatch 2024-03-11T22:22:36.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]: dispatch 2024-03-11T22:22:36.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8dc26f0-417a-4358-9dd1-c103fe3caf02"}]': finished 2024-03-11T22:22:36.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T22:22:36.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:36.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:36 smithi073 ceph-mon[24794]: from='client.? 172.21.15.99:0/877359255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:22:38.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:38 smithi099 ceph-mon[27157]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:38.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:38 smithi073 ceph-mon[24794]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:40.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:40 smithi099 ceph-mon[27157]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:40.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:40 smithi073 ceph-mon[24794]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:41.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:22:41.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:22:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:22:41.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:22:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:42 smithi099 ceph-mon[27157]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:22:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:42 smithi099 ceph-mon[27157]: Deploying daemon osd.7 on smithi099 2024-03-11T22:22:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:42 smithi073 ceph-mon[24794]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:22:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:42.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:42 smithi073 ceph-mon[24794]: Deploying daemon osd.7 on smithi099 2024-03-11T22:22:44.765 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:44 smithi099 ceph-mon[27157]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:44.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:44 smithi073 ceph-mon[24794]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:46.600 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:46 smithi099 ceph-mon[27157]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:46.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:46 smithi073 ceph-mon[24794]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:48.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:48.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:48.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:48.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:48.225 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:48.226 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:48.226 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:48.226 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:49.197 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:48 smithi099 ceph-mon[27157]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:49.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:48 smithi073 ceph-mon[24794]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:50.745 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 7 on host 'smithi099' 2024-03-11T22:22:50.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:50 smithi099 ceph-mon[27157]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:50.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:50 smithi099 ceph-mon[27157]: from='osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:22:50.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:50 smithi099 ceph-mon[27157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:22:50.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:50 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:51.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:50 smithi073 ceph-mon[24794]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:51.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:50 smithi073 ceph-mon[24794]: from='osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:22:51.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:50 smithi073 ceph-mon[24794]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:22:51.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:50 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:51.365 DEBUG:teuthology.orchestra.run.smithi099:osd.7> sudo journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T22:22:51.367 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T22:22:51.367 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:22:51.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:22:51.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: osdmap e54: 8 total, 7 up, 8 in 2024-03-11T22:22:51.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:51.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: from='osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:51.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:51.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:51 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:52.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:22:52.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: osdmap e54: 8 total, 7 up, 8 in 2024-03-11T22:22:52.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:52.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: from='osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:52.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:22:52.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:52.172 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[40418]: 2024-03-11T22:22:51.870+0000 7fdee733b700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T22:22:52.172 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:22:51 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[40418]: 2024-03-11T22:22:51.878+0000 7fdee24d3700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:22:52.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:52 smithi073 ceph-mon[24794]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:52.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:52 smithi073 ceph-mon[24794]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:52.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:52 smithi073 ceph-mon[24794]: osdmap e55: 8 total, 7 up, 8 in 2024-03-11T22:22:52.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:52 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:52.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:52 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:53.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:52 smithi099 ceph-mon[27157]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:22:53.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:52 smithi099 ceph-mon[27157]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-03-11T22:22:53.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:52 smithi099 ceph-mon[27157]: osdmap e55: 8 total, 7 up, 8 in 2024-03-11T22:22:53.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:52 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:53.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:52 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:53.334 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:22:53.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: purged_snaps scrub starts 2024-03-11T22:22:53.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: purged_snaps scrub ok 2024-03-11T22:22:53.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203] boot 2024-03-11T22:22:53.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T22:22:53.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:53.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2653219253' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:22:53.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:53 smithi073 ceph-mon[24794]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T22:22:54.016 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1710195772,"num_in_osds":8,"osd_in_since":1710195755,"num_remapped_pgs":1} 2024-03-11T22:22:54.016 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: purged_snaps scrub starts 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: purged_snaps scrub ok 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: osd.7 [v2:172.21.15.99:6824/3143512203,v1:172.21.15.99:6825/3143512203] boot 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2653219253' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:22:54.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:53 smithi099 ceph-mon[27157]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T22:22:54.239 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:54.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:54 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:54.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T22:22:54.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:54 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:22:56.016 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:22:56.016 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":59,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:16:52.064669+0000","modified":"2024-03-11T22:22:55.344561+0000","last_up_change":"2024-03-11T22:22:52.864870+0000","last_in_change":"2024-03-11T22:22:35.517636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:20:55.282438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ade19a11-36e5-4623-bd98-f6e83bd1ae95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6803","nonce":296531743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6805","nonce":296531743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6809","nonce":296531743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6807","nonce":296531743}]},"public_addr":"172.21.15.73:6803/296531743","cluster_addr":"172.21.15.73:6805/296531743","heartbeat_back_addr":"172.21.15.73:6809/296531743","heartbeat_front_addr":"172.21.15.73:6807/296531743","state":["exists","up"]},{"osd":1,"uuid":"8ead05ba-132c-4a2d-b442-95a6ed3ce29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6811","nonce":1931878266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6813","nonce":1931878266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6817","nonce":1931878266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6815","nonce":1931878266}]},"public_addr":"172.21.15.73:6811/1931878266","cluster_addr":"172.21.15.73:6813/1931878266","heartbeat_back_addr":"172.21.15.73:6817/1931878266","heartbeat_front_addr":"172.21.15.73:6815/1931878266","state":["exists","up"]},{"osd":2,"uuid":"cef61f19-bf1e-4f37-b2dc-c33ca51b85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6819","nonce":2023932795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6821","nonce":2023932795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6825","nonce":2023932795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6823","nonce":2023932795}]},"public_addr":"172.21.15.73:6819/2023932795","cluster_addr":"172.21.15.73:6821/2023932795","heartbeat_back_addr":"172.21.15.73:6825/2023932795","heartbeat_front_addr":"172.21.15.73:6823/2023932795","state":["exists","up"]},{"osd":3,"uuid":"2aa5a768-6564-4c73-9414-06634f937eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6827","nonce":1137804667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6829","nonce":1137804667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6833","nonce":1137804667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6831","nonce":1137804667}]},"public_addr":"172.21.15.73:6827/1137804667","cluster_addr":"172.21.15.73:6829/1137804667","heartbeat_back_addr":"172.21.15.73:6833/1137804667","heartbeat_front_addr":"172.21.15.73:6831/1137804667","state":["exists","up"]},{"osd":4,"uuid":"d27896cd-413d-4fb3-832e-5b1c25b95dec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6801","nonce":2864491863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6803","nonce":2864491863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6807","nonce":2864491863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6805","nonce":2864491863}]},"public_addr":"172.21.15.99:6801/2864491863","cluster_addr":"172.21.15.99:6803/2864491863","heartbeat_back_addr":"172.21.15.99:6807/2864491863","heartbeat_front_addr":"172.21.15.99:6805/2864491863","state":["exists","up"]},{"osd":5,"uuid":"21863417-f4e5-4f0e-b42c-edcf00f14a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6809","nonce":390666098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6811","nonce":390666098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6815","nonce":390666098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6813","nonce":390666098}]},"public_addr":"172.21.15.99:6809/390666098","cluster_addr":"172.21.15.99:6811/390666098","heartbeat_back_addr":"172.21.15.99:6815/390666098","heartbeat_front_addr":"172.21.15.99:6813/390666098","state":["exists","up"]},{"osd":6,"uuid":"8f9d2ebd-1391-451a-a6f6-99e279862cf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6817","nonce":3496847709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6819","nonce":3496847709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6823","nonce":3496847709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6821","nonce":3496847709}]},"public_addr":"172.21.15.99:6817/3496847709","cluster_addr":"172.21.15.99:6819/3496847709","heartbeat_back_addr":"172.21.15.99:6823/3496847709","heartbeat_front_addr":"172.21.15.99:6821/3496847709","state":["exists","up"]},{"osd":7,"uuid":"c8dc26f0-417a-4358-9dd1-c103fe3caf02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6825","nonce":3143512203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6827","nonce":3143512203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6831","nonce":3143512203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6829","nonce":3143512203}]},"public_addr":"172.21.15.99:6825/3143512203","cluster_addr":"172.21.15.99:6827/3143512203","heartbeat_back_addr":"172.21.15.99:6831/3143512203","heartbeat_front_addr":"172.21.15.99:6829/3143512203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:52.960539+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:25.355524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:51.428622+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:17.249765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:38.717423+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:02.866063+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:26.426439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:50.978430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2530856067":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2389307632":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/4196696409":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/758380471":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/3222669240":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/2937294181":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:6800/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/3444768794":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/93711661":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/2409839089":"2024-03-12T22:17:16.046923+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:22:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:55 smithi073 ceph-mon[24794]: Detected new or changed devices on smithi099 2024-03-11T22:22:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:55 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi099 to 3984M 2024-03-11T22:22:56.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:55 smithi073 ceph-mon[24794]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T22:22:56.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:55 smithi099 ceph-mon[27157]: Detected new or changed devices on smithi099 2024-03-11T22:22:56.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:55 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi099 to 3984M 2024-03-11T22:22:56.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:55 smithi099 ceph-mon[27157]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T22:22:56.538 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:20:55.282438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-11T22:22:56.538 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:22:56.753 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:22:57.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:56 smithi099 ceph-mon[27157]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:57.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:56 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/237849379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:22:57.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:56 smithi099 ceph-mon[27157]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T22:22:57.160 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:56 smithi073 ceph-mon[24794]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:57.160 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:56 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/237849379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:22:57.160 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:56 smithi073 ceph-mon[24794]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T22:22:58.437 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-11T22:22:59.130 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:22:59.130 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:22:59.147 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:58 smithi073 ceph-mon[24794]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:59.147 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:22:58 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/49670453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:22:59.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:58 smithi099 ceph-mon[27157]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:22:59.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:22:58 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/49670453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:23:01.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:00 smithi073 ceph-mon[24794]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T22:23:01.227 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2024-03-11T22:23:01.227 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBFhO9l/r04DRAAnWj+WrBLBIBMijMRCYF/Sg== 2024-03-11T22:23:01.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:00 smithi099 ceph-mon[27157]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T22:23:01.895 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-11T22:23:01.896 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:23:01.896 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:23:01.980 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:23:01.980 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:23:01.980 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:23:02.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:01 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2070886332' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:23:02.215 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:01 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2070886332' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:23:02.230 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2070886332' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:23:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:01 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2070886332' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:23:02.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:02 smithi073 ceph-mon[24794]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T22:23:03.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:02 smithi099 ceph-mon[27157]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-11T22:23:04.042 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:04.669 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:04 smithi073 ceph-mon[24794]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T22:23:04.669 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:04 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1473052744' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:23:04.695 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi073.pglkvm","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":1426395177},{"type":"v1","addr":"172.21.15.73:6801","nonce":1426395177}]},"active_addr":"172.21.15.73:6801/1426395177","active_change":"2024-03-11T22:18:40.649991+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi099.rcoixq","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.73:8443/","prometheus":"http://172.21.15.73:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":2458930339}]},{"addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1703963737}]},{"addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":2327315065}]},{"addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":654186106}]}]}} 2024-03-11T22:23:04.701 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:23:04.701 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:23:04.701 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:23:04.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:04 smithi099 ceph-mon[27157]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T22:23:04.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:04 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1473052744' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:23:04.918 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:06.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:06 smithi073 ceph-mon[24794]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-11T22:23:06.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:06 smithi099 ceph-mon[27157]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-11T22:23:06.765 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:06.765 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":60,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:16:52.064669+0000","modified":"2024-03-11T22:22:56.345963+0000","last_up_change":"2024-03-11T22:22:52.864870+0000","last_in_change":"2024-03-11T22:22:35.517636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:20:55.282438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ade19a11-36e5-4623-bd98-f6e83bd1ae95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6803","nonce":296531743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6805","nonce":296531743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6809","nonce":296531743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6807","nonce":296531743}]},"public_addr":"172.21.15.73:6803/296531743","cluster_addr":"172.21.15.73:6805/296531743","heartbeat_back_addr":"172.21.15.73:6809/296531743","heartbeat_front_addr":"172.21.15.73:6807/296531743","state":["exists","up"]},{"osd":1,"uuid":"8ead05ba-132c-4a2d-b442-95a6ed3ce29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6811","nonce":1931878266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6813","nonce":1931878266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6817","nonce":1931878266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6815","nonce":1931878266}]},"public_addr":"172.21.15.73:6811/1931878266","cluster_addr":"172.21.15.73:6813/1931878266","heartbeat_back_addr":"172.21.15.73:6817/1931878266","heartbeat_front_addr":"172.21.15.73:6815/1931878266","state":["exists","up"]},{"osd":2,"uuid":"cef61f19-bf1e-4f37-b2dc-c33ca51b85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6819","nonce":2023932795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6821","nonce":2023932795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6825","nonce":2023932795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6823","nonce":2023932795}]},"public_addr":"172.21.15.73:6819/2023932795","cluster_addr":"172.21.15.73:6821/2023932795","heartbeat_back_addr":"172.21.15.73:6825/2023932795","heartbeat_front_addr":"172.21.15.73:6823/2023932795","state":["exists","up"]},{"osd":3,"uuid":"2aa5a768-6564-4c73-9414-06634f937eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6827","nonce":1137804667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6829","nonce":1137804667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6833","nonce":1137804667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6831","nonce":1137804667}]},"public_addr":"172.21.15.73:6827/1137804667","cluster_addr":"172.21.15.73:6829/1137804667","heartbeat_back_addr":"172.21.15.73:6833/1137804667","heartbeat_front_addr":"172.21.15.73:6831/1137804667","state":["exists","up"]},{"osd":4,"uuid":"d27896cd-413d-4fb3-832e-5b1c25b95dec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6801","nonce":2864491863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6803","nonce":2864491863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6807","nonce":2864491863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6805","nonce":2864491863}]},"public_addr":"172.21.15.99:6801/2864491863","cluster_addr":"172.21.15.99:6803/2864491863","heartbeat_back_addr":"172.21.15.99:6807/2864491863","heartbeat_front_addr":"172.21.15.99:6805/2864491863","state":["exists","up"]},{"osd":5,"uuid":"21863417-f4e5-4f0e-b42c-edcf00f14a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6809","nonce":390666098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6811","nonce":390666098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6815","nonce":390666098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6813","nonce":390666098}]},"public_addr":"172.21.15.99:6809/390666098","cluster_addr":"172.21.15.99:6811/390666098","heartbeat_back_addr":"172.21.15.99:6815/390666098","heartbeat_front_addr":"172.21.15.99:6813/390666098","state":["exists","up"]},{"osd":6,"uuid":"8f9d2ebd-1391-451a-a6f6-99e279862cf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6817","nonce":3496847709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6819","nonce":3496847709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6823","nonce":3496847709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6821","nonce":3496847709}]},"public_addr":"172.21.15.99:6817/3496847709","cluster_addr":"172.21.15.99:6819/3496847709","heartbeat_back_addr":"172.21.15.99:6823/3496847709","heartbeat_front_addr":"172.21.15.99:6821/3496847709","state":["exists","up"]},{"osd":7,"uuid":"c8dc26f0-417a-4358-9dd1-c103fe3caf02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6825","nonce":3143512203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6827","nonce":3143512203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6831","nonce":3143512203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6829","nonce":3143512203}]},"public_addr":"172.21.15.99:6825/3143512203","cluster_addr":"172.21.15.99:6827/3143512203","heartbeat_back_addr":"172.21.15.99:6831/3143512203","heartbeat_front_addr":"172.21.15.99:6829/3143512203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:52.960539+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:25.355524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:51.428622+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:17.249765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:38.717423+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:02.866063+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:26.426439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:50.978430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2530856067":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2389307632":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/4196696409":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/758380471":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/3222669240":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/2937294181":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:6800/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/3444768794":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/93711661":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/2409839089":"2024-03-12T22:17:16.046923+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:23:07.458 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:23:07.459 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:23:07.670 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:07.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:07 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1314816904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:23:07.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:07 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1314816904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:23:08.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:08 smithi073 ceph-mon[24794]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T22:23:08.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:08 smithi099 ceph-mon[27157]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-11T22:23:09.355 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:09.356 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":60,"fsid":"f2f121a8-dff4-11ee-95c8-87774f69a715","created":"2024-03-11T22:16:52.064669+0000","modified":"2024-03-11T22:22:56.345963+0000","last_up_change":"2024-03-11T22:22:52.864870+0000","last_in_change":"2024-03-11T22:22:35.517636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:20:55.282438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ade19a11-36e5-4623-bd98-f6e83bd1ae95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6803","nonce":296531743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6805","nonce":296531743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6809","nonce":296531743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":296531743},{"type":"v1","addr":"172.21.15.73:6807","nonce":296531743}]},"public_addr":"172.21.15.73:6803/296531743","cluster_addr":"172.21.15.73:6805/296531743","heartbeat_back_addr":"172.21.15.73:6809/296531743","heartbeat_front_addr":"172.21.15.73:6807/296531743","state":["exists","up"]},{"osd":1,"uuid":"8ead05ba-132c-4a2d-b442-95a6ed3ce29a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6811","nonce":1931878266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6813","nonce":1931878266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6817","nonce":1931878266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":1931878266},{"type":"v1","addr":"172.21.15.73:6815","nonce":1931878266}]},"public_addr":"172.21.15.73:6811/1931878266","cluster_addr":"172.21.15.73:6813/1931878266","heartbeat_back_addr":"172.21.15.73:6817/1931878266","heartbeat_front_addr":"172.21.15.73:6815/1931878266","state":["exists","up"]},{"osd":2,"uuid":"cef61f19-bf1e-4f37-b2dc-c33ca51b85ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6819","nonce":2023932795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6821","nonce":2023932795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6825","nonce":2023932795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2023932795},{"type":"v1","addr":"172.21.15.73:6823","nonce":2023932795}]},"public_addr":"172.21.15.73:6819/2023932795","cluster_addr":"172.21.15.73:6821/2023932795","heartbeat_back_addr":"172.21.15.73:6825/2023932795","heartbeat_front_addr":"172.21.15.73:6823/2023932795","state":["exists","up"]},{"osd":3,"uuid":"2aa5a768-6564-4c73-9414-06634f937eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6827","nonce":1137804667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6829","nonce":1137804667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6833","nonce":1137804667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1137804667},{"type":"v1","addr":"172.21.15.73:6831","nonce":1137804667}]},"public_addr":"172.21.15.73:6827/1137804667","cluster_addr":"172.21.15.73:6829/1137804667","heartbeat_back_addr":"172.21.15.73:6833/1137804667","heartbeat_front_addr":"172.21.15.73:6831/1137804667","state":["exists","up"]},{"osd":4,"uuid":"d27896cd-413d-4fb3-832e-5b1c25b95dec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6801","nonce":2864491863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6803","nonce":2864491863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6807","nonce":2864491863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2864491863},{"type":"v1","addr":"172.21.15.99:6805","nonce":2864491863}]},"public_addr":"172.21.15.99:6801/2864491863","cluster_addr":"172.21.15.99:6803/2864491863","heartbeat_back_addr":"172.21.15.99:6807/2864491863","heartbeat_front_addr":"172.21.15.99:6805/2864491863","state":["exists","up"]},{"osd":5,"uuid":"21863417-f4e5-4f0e-b42c-edcf00f14a0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6809","nonce":390666098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6811","nonce":390666098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6815","nonce":390666098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":390666098},{"type":"v1","addr":"172.21.15.99:6813","nonce":390666098}]},"public_addr":"172.21.15.99:6809/390666098","cluster_addr":"172.21.15.99:6811/390666098","heartbeat_back_addr":"172.21.15.99:6815/390666098","heartbeat_front_addr":"172.21.15.99:6813/390666098","state":["exists","up"]},{"osd":6,"uuid":"8f9d2ebd-1391-451a-a6f6-99e279862cf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6817","nonce":3496847709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6819","nonce":3496847709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6823","nonce":3496847709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":3496847709},{"type":"v1","addr":"172.21.15.99:6821","nonce":3496847709}]},"public_addr":"172.21.15.99:6817/3496847709","cluster_addr":"172.21.15.99:6819/3496847709","heartbeat_back_addr":"172.21.15.99:6823/3496847709","heartbeat_front_addr":"172.21.15.99:6821/3496847709","state":["exists","up"]},{"osd":7,"uuid":"c8dc26f0-417a-4358-9dd1-c103fe3caf02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6825","nonce":3143512203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6827","nonce":3143512203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6831","nonce":3143512203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3143512203},{"type":"v1","addr":"172.21.15.99:6829","nonce":3143512203}]},"public_addr":"172.21.15.99:6825/3143512203","cluster_addr":"172.21.15.99:6827/3143512203","heartbeat_back_addr":"172.21.15.99:6831/3143512203","heartbeat_front_addr":"172.21.15.99:6829/3143512203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:19:52.960539+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:25.355524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:20:51.428622+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:17.249765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:21:38.717423+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:02.866063+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:26.426439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:22:50.978430+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/545345264":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2530856067":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/2389307632":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/4196696409":"2024-03-12T22:18:40.649686+0000","172.21.15.73:6800/1932167363":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/758380471":"2024-03-12T22:18:40.649686+0000","172.21.15.73:0/3222669240":"2024-03-12T22:17:44.825855+0000","172.21.15.73:0/2937294181":"2024-03-12T22:17:44.825855+0000","172.21.15.73:6801/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:6800/2138389529":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/3444768794":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/93711661":"2024-03-12T22:17:16.046923+0000","172.21.15.73:0/2409839089":"2024-03-12T22:17:16.046923+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:23:09.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:09 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/765420004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:23:09.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:09 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/765420004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:23:09.991 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:23:09.992 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:23:09.992 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:23:09.993 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T22:23:09.993 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T22:23:09.993 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T22:23:09.994 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T22:23:09.994 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T22:23:10.426 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.426 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.427 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.427 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.429 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.493 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:10.675 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:10 smithi073 ceph-mon[24794]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-11T22:23:10.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:10 smithi099 ceph-mon[27157]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-03-11T22:23:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:12.535 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:12 smithi073 ceph-mon[24794]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:12.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:12 smithi099 ceph-mon[27157]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:14.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:14 smithi073 ceph-mon[24794]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:14.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:14 smithi099 ceph-mon[27157]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:16.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:16 smithi073 ceph-mon[24794]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:16.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:16 smithi099 ceph-mon[27157]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:18.708 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:18 smithi073 ceph-mon[24794]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:18.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:18 smithi099 ceph-mon[27157]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:20.277 INFO:teuthology.orchestra.run.smithi073.stdout:171798691857 2024-03-11T22:23:20.277 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T22:23:20.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:20 smithi073 ceph-mon[24794]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:20.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:20 smithi099 ceph-mon[27157]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:21.701 INFO:teuthology.orchestra.run.smithi073.stdout:240518168584 2024-03-11T22:23:21.701 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T22:23:22.381 INFO:teuthology.orchestra.run.smithi073.stdout:60129542182 2024-03-11T22:23:22.381 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:23:22.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:22 smithi073 ceph-mon[24794]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:22.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:22 smithi099 ceph-mon[27157]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:24.221 INFO:teuthology.orchestra.run.smithi073.stdout:206158430222 2024-03-11T22:23:24.222 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T22:23:24.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:24 smithi073 ceph-mon[24794]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:24.663 INFO:teuthology.orchestra.run.smithi073.stdout:38654705709 2024-03-11T22:23:24.663 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:23:24.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:24 smithi099 ceph-mon[27157]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:26.103 INFO:teuthology.orchestra.run.smithi073.stdout:137438953494 2024-03-11T22:23:26.103 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T22:23:26.138 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:26.139 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:26.139 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:26.139 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:26.173 INFO:teuthology.orchestra.run.smithi073.stdout:115964117020 2024-03-11T22:23:26.173 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T22:23:26.377 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:26.631 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:26 smithi073 ceph-mon[24794]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:26.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:26 smithi099 ceph-mon[27157]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:27.796 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:28.568 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:28 smithi073 ceph-mon[24794]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:28.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:28 smithi099 ceph-mon[27157]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:29.260 INFO:teuthology.orchestra.run.smithi073.stdout:38654705710 2024-03-11T22:23:29.681 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:29 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2782009279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:23:29.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:29 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2782009279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:23:30.257 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:30.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:30 smithi073 ceph-mon[24794]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:30.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:30 smithi099 ceph-mon[27157]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:32.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:32 smithi073 ceph-mon[24794]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:32.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:32 smithi099 ceph-mon[27157]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:32.928 INFO:teuthology.orchestra.run.smithi073.stdout:206158430223 2024-03-11T22:23:33.439 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:33 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1104630099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:23:33.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:33 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1104630099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:23:34.388 INFO:teuthology.orchestra.run.smithi073.stdout:81604378658 2024-03-11T22:23:34.389 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:23:34.656 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:34 smithi073 ceph-mon[24794]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:34.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:34 smithi099 ceph-mon[27157]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:35.347 INFO:teuthology.orchestra.run.smithi073.stdout:60129542184 2024-03-11T22:23:35.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:35 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1612828281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:23:35.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:35 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1612828281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:23:35.848 INFO:teuthology.orchestra.run.smithi073.stdout:171798691861 2024-03-11T22:23:36.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691857 got 171798691861 for osd.5 2024-03-11T22:23:36.425 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:36.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:36 smithi073 ceph-mon[24794]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:36.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:36 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2543658190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:23:36.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:36 smithi099 ceph-mon[27157]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:36 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2543658190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:23:36.789 INFO:teuthology.orchestra.run.smithi073.stdout:137438953497 2024-03-11T22:23:37.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430223 for osd.6 2024-03-11T22:23:37.002 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:37.460 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3511609939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:23:37.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:37 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3511609939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:23:37.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542184 for osd.1 2024-03-11T22:23:37.754 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:38.127 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:38.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705710 for osd.0 2024-03-11T22:23:38.137 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:38.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:38 smithi073 ceph-mon[24794]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:38.717 INFO:teuthology.orchestra.run.smithi073.stdout:115964117023 2024-03-11T22:23:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:38 smithi099 ceph-mon[27157]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:39.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:39 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1417200269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:23:39.639 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:39 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1417200269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:23:40.099 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117020 got 115964117023 for osd.3 2024-03-11T22:23:40.099 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:40.537 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953494 got 137438953497 for osd.4 2024-03-11T22:23:40.537 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:40.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:40 smithi073 ceph-mon[24794]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:40.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:40 smithi099 ceph-mon[27157]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:41.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:23:41.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:23:41.482 INFO:teuthology.orchestra.run.smithi073.stdout:81604378661 2024-03-11T22:23:41.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:23:41.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:23:42.041 INFO:teuthology.orchestra.run.smithi073.stdout:240518168588 2024-03-11T22:23:42.249 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378661 for osd.2 2024-03-11T22:23:42.249 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:42.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:42 smithi073 ceph-mon[24794]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:42.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:42 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/218636373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:23:42.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:42 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/4227666732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:23:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:42 smithi099 ceph-mon[27157]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:42 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/218636373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:23:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:42 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/4227666732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:23:42.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168584 got 240518168588 for osd.7 2024-03-11T22:23:42.929 DEBUG:teuthology.parallel:result is None 2024-03-11T22:23:42.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:23:42.930 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:23:43.142 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:44.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:44 smithi073 ceph-mon[24794]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:44.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:44 smithi099 ceph-mon[27157]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:44.761 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:44.762 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-11T22:23:45.376 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-11T22:23:43.010435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49128,"kb_used_data":6376,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912088,"statfs":{"total":767960285184,"available":767909978112,"internally_reserved":0,"allocated":6529024,"data_stored":3571686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002032"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-11T22:22:56.355364+0000","last_change":"2024-03-11T22:22:56.355364+0000","last_active":"2024-03-11T22:22:56.355364+0000","last_peered":"2024-03-11T22:22:56.355364+0000","last_clean":"2024-03-11T22:22:56.355364+0000","last_became_active":"2024-03-11T22:22:56.355005+0000","last_became_peered":"2024-03-11T22:22:56.355005+0000","last_unstale":"2024-03-11T22:22:56.355364+0000","last_undegraded":"2024-03-11T22:22:56.355364+0000","last_fullsized":"2024-03-11T22:22:56.355364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:55.369759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:55.369759+0000","last_clean_scrub_stamp":"2024-03-11T22:20:55.369759+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:13.306568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":6,"up_from":48,"seq":206158430225,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52700000000000002}]},{"osd":2,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.54300000000000004}]},{"osd":3,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.63100000000000001}]},{"osd":4,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.70799999999999996}]},{"osd":5,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27400000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46400000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.375}]},{"osd":2,"last update":"Mon Mar 11 22:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38,"15min":0.377},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.374},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.318}]},{"osd":3,"last update":"Mon Mar 11 22:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53400000000000003,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":5.5830000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54000000000000004}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":5.5519999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6956,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738196,"statfs":{"total":95995035648,"available":95987912704,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 22:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.33500000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.32900000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.48499999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 11 22:22:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.379,"15min":0.37},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39800000000000002,"15min":0.38800000000000001},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.503}]},{"osd":3,"last update":"Mon Mar 11 22:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.45000000000000001,"15min":0.45600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":0.47199999999999998,"15min":0.47799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58899999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Mon Mar 11 22:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.90800000000000003}]},{"osd":5,"last update":"Mon Mar 11 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998}]},{"osd":6,"last update":"Mon Mar 11 22:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.876}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.377,"15min":0.372},"min":{"1min":0.26500000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39100000000000001}]},{"osd":1,"last update":"Mon Mar 11 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.311}]},{"osd":3,"last update":"Mon Mar 11 22:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45100000000000001},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.29799999999999999}]},{"osd":4,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon Mar 11 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.627,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.437,"15min":0.433},"min":{"1min":0.32900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.438,"15min":0.436},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5792,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739360,"statfs":{"total":95995035648,"available":95989104640,"internally_reserved":0,"allocated":614400,"data_stored":248002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.499}]},{"osd":1,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.627},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 11 22:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.38500000000000001}]},{"osd":6,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.44600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":5,"up_from":40,"seq":171798691862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":247621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":1,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63}]},{"osd":3,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 11 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:23:45.378 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:23:45.598 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:45.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:45 smithi073 ceph-mon[24794]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:23:45.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:45 smithi099 ceph-mon[27157]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:23:46.644 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:46 smithi073 ceph-mon[24794]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:46.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:46 smithi099 ceph-mon[27157]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:47.321 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:47.322 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-11T22:23:48.082 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-11T22:23:47.011286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49128,"kb_used_data":6376,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912088,"statfs":{"total":767960285184,"available":767909978112,"internally_reserved":0,"allocated":6529024,"data_stored":3571686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002200"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-11T22:22:56.355364+0000","last_change":"2024-03-11T22:22:56.355364+0000","last_active":"2024-03-11T22:22:56.355364+0000","last_peered":"2024-03-11T22:22:56.355364+0000","last_clean":"2024-03-11T22:22:56.355364+0000","last_became_active":"2024-03-11T22:22:56.355005+0000","last_became_peered":"2024-03-11T22:22:56.355005+0000","last_unstale":"2024-03-11T22:22:56.355364+0000","last_undegraded":"2024-03-11T22:22:56.355364+0000","last_fullsized":"2024-03-11T22:22:56.355364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:55.369759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:55.369759+0000","last_clean_scrub_stamp":"2024-03-11T22:20:55.369759+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:13.306568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":6,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.60599999999999998}]},{"osd":1,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.502}]},{"osd":4,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.435},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35399999999999998,"15min":0.34200000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.27400000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.46400000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Mar 11 22:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38,"15min":0.377},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38,"15min":0.374},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Mar 11 22:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53400000000000003,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.245},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44800000000000001,"15min":0.45600000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.308}]},{"osd":4,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6956,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738196,"statfs":{"total":95995035648,"available":95987912704,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 22:23:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.33500000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.499,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.255},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.32900000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.48499999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.26600000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:22:59 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.379,"15min":0.37},"min":{"1min":0.29999999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39800000000000002,"15min":0.38800000000000001},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Mar 11 22:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.45000000000000001,"15min":0.45600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.47199999999999998,"15min":0.47799999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58899999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Mar 11 22:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.70599999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90800000000000003,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Mar 11 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.625},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Mar 11 22:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6308,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.377,"15min":0.372},"min":{"1min":0.26500000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Mar 11 22:22:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Mar 11 22:23:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45100000000000001},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.318},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:22:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Mar 11 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.44500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.55000000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34200000000000003}]},{"osd":1,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.627,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.437,"15min":0.433},"min":{"1min":0.32900000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Mar 11 22:23:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.438,"15min":0.436},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 11 22:22:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Mar 11 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.434},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.50800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5792,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739360,"statfs":{"total":95995035648,"available":95989104640,"internally_reserved":0,"allocated":614400,"data_stored":248002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.44900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65800000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Mar 11 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.65100000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46400000000000002}]},{"osd":2,"last update":"Mon Mar 11 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71099999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.42699999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.71499999999999997,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Mar 11 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55200000000000005,"15min":0.54800000000000004},"min":{"1min":0.48399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.46400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.74299999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Mar 11 22:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Mar 11 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":247621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.622},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Mar 11 22:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Mar 11 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.64900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:23:48.084 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:23:48.084 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:23:48.084 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:23:48.084 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:23:48.297 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/mon.smithi073/config 2024-03-11T22:23:48.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:48 smithi073 ceph-mon[24794]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:48.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:48 smithi073 ceph-mon[24794]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:23:48.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:48 smithi099 ceph-mon[27157]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:48.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:48 smithi099 ceph-mon[27157]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:23:50.100 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:23:50.101 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:23:50.715 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:50 smithi073 ceph-mon[24794]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:50.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:50 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2734480175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:23:50.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:50 smithi099 ceph-mon[27157]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:50.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:50 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2734480175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:23:50.745 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:23:50.746 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:23:50.746 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:23:50.755 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:23:50.755 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T22:23:52.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:52 smithi073 ceph-mon[24794]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:52.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:52 smithi099 ceph-mon[27157]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:52.758 INFO:teuthology.orchestra.run.smithi073.stdout:Backend: cephadm 2024-03-11T22:23:52.758 INFO:teuthology.orchestra.run.smithi073.stdout:Available: Yes 2024-03-11T22:23:52.758 INFO:teuthology.orchestra.run.smithi073.stdout:Paused: No 2024-03-11T22:23:53.361 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:23:53.667 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:53 smithi073 ceph-mon[24794]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:23:53.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:53 smithi099 ceph-mon[27157]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:23:54.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:54 smithi099 ceph-mon[27157]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:54.776 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:54 smithi073 ceph-mon[24794]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:55.162 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (4m) 2m ago 5m 14.1M - ba2b418f427c e10c29c255a9 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (5m) 2m ago 5m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (5m) 63s ago 5m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (4m) 2m ago 5m 44.9M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (4m) 63s ago 4m 411M - 17.2.0 e1d6a67b021e 824e312fca3c 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (6m) 2m ago 7m 42.0M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (4m) 63s ago 4m 39.3M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (5m) 2m ago 5m 17.2M - 1dbe0e931976 46f528d06727 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (4m) 63s ago 4m 16.0M - 1dbe0e931976 fb9193a63c59 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (4m) 2m ago 4m 34.1M 3216M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (3m) 2m ago 3m 37.1M 3216M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (3m) 2m ago 3m 33.0M 3216M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:23:55.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (2m) 2m ago 2m 12.0M 3216M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:23:55.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (2m) 63s ago 2m 34.7M 3984M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:23:55.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (116s) 63s ago 116s 32.4M 3984M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:23:55.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (93s) 63s ago 92s 30.1M 3984M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:23:55.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (68s) 63s ago 68s 10.4M 3984M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:23:55.164 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (4m) 2m ago 4m 51.7M - 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:23:55.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:23:55.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:23:55.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:23:55.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:55 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:23:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:23:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:23:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:23:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:55 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:23:55.741 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:23:57.095 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:56 smithi073 ceph-mon[24794]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:57.095 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:56 smithi073 ceph-mon[24794]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:23:57.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:56 smithi099 ceph-mon[27157]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:57.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:56 smithi099 ceph-mon[27157]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:23:57.598 INFO:teuthology.orchestra.run.smithi073.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:23:57.598 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:crash 2/2 2m ago 6m * 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:mgr 2/2 2m ago 6m count:2 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:mon 2/2 2m ago 5m smithi073:172.21.15.73=smithi073;smithi099:172.21.15.99=smithi099;count:2 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:osd 8 2m ago - 2024-03-11T22:23:57.599 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-11T22:23:58.237 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:23:59.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:58 smithi073 ceph-mon[24794]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:59.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:23:58 smithi073 ceph-mon[24794]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:23:59.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:58 smithi099 ceph-mon[27157]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:23:59.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:23:58 smithi099 ceph-mon[27157]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:00.092 INFO:teuthology.orchestra.run.smithi073.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:24:00.092 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 172.21.15.73 2024-03-11T22:24:00.092 INFO:teuthology.orchestra.run.smithi073.stdout:smithi099 172.21.15.99 2024-03-11T22:24:00.092 INFO:teuthology.orchestra.run.smithi073.stdout:2 hosts in cluster 2024-03-11T22:24:00.733 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:24:00.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:00 smithi073 ceph-mon[24794]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:01.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:00 smithi099 ceph-mon[27157]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:02.139 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:01 smithi073 ceph-mon[24794]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:02.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:01 smithi099 ceph-mon[27157]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:02.644 INFO:teuthology.orchestra.run.smithi073.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-03-11T22:24:02.644 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040115400BGN 400G LVM detected, locked 2024-03-11T22:24:02.644 INFO:teuthology.orchestra.run.smithi073.stdout:smithi099 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040101400BGN 400G LVM detected, locked 2024-03-11T22:24:03.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:02 smithi073 ceph-mon[24794]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:03.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:02 smithi099 ceph-mon[27157]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:03.357 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-11T22:24:03.366 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:03.366 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-11T22:24:03.395 INFO:teuthology.orchestra.run.smithi073.stderr:+ systemctl stop nfs-server 2024-03-11T22:24:03.403 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi099.front.sepia.ceph.com 2024-03-11T22:24:03.403 DEBUG:teuthology.orchestra.run.smithi099:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-11T22:24:03.435 INFO:teuthology.orchestra.run.smithi099.stderr:+ systemctl stop nfs-server 2024-03-11T22:24:03.444 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:24:03.451 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:03.451 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-03-11T22:24:04.051 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:03 smithi073 ceph-mon[24794]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:04.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:03 smithi099 ceph-mon[27157]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:05.098 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:04 smithi073 ceph-mon[24794]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:04 smithi099 ceph-mon[27157]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:06.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:05 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-11T22:24:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:05 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-11T22:24:07.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-mon[24794]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:07.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-mon[24794]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:07.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-11T22:24:07.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-mon[24794]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T22:24:07.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-11T22:24:07.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:06 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[24790]: 2024-03-11T22:24:06.911+0000 7fbd7cb2c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:24:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:06 smithi099 ceph-mon[27157]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:06 smithi099 ceph-mon[27157]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:06 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-11T22:24:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:06 smithi099 ceph-mon[27157]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T22:24:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:06 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-11T22:24:07.613 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-03-11T22:24:07.622 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-03-11T22:24:07.622 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch ls -f json 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-11T22:24:08.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: fsmap foofs:0 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: Saving service mds.foofs spec with placement count:2 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: Deploying daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:24:08.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:07 smithi073 ceph-mon[24794]: pgmap v214: 65 pgs: 1 creating+peering, 1 active+clean, 63 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: fsmap foofs:0 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: Saving service mds.foofs spec with placement count:2 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:08.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: Deploying daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:24:08.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:07 smithi099 ceph-mon[27157]: pgmap v214: 65 pgs: 1 creating+peering, 1 active+clean, 63 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:09.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:08 smithi073 ceph-mon[24794]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T22:24:09.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:08 smithi099 ceph-mon[27157]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T22:24:10.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:09 smithi073 ceph-mon[24794]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T22:24:10.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:09 smithi073 ceph-mon[24794]: pgmap v217: 65 pgs: 21 active+clean, 2 creating+activating, 7 creating+peering, 35 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:10.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:09 smithi099 ceph-mon[27157]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T22:24:10.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:09 smithi099 ceph-mon[27157]: pgmap v217: 65 pgs: 21 active+clean, 2 creating+activating, 7 creating+peering, 35 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:10.848 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:24:10.848 INFO:teuthology.orchestra.run.smithi073.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-11T22:17:33.673110Z", "last_refresh": "2024-03-11T22:21:19.892766Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-11T22:17:28.345495Z", "last_refresh": "2024-03-11T22:21:19.892843Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-11T22:17:31.125131Z", "last_refresh": "2024-03-11T22:21:19.892907Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-11T22:24:06.933051Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-11T22:24:06.925899Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-11T22:17:26.767881Z", "last_refresh": "2024-03-11T22:21:19.892696Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi073:172.21.15.73=smithi073", "smithi099:172.21.15.99=smithi099"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-11T22:18:16.830889Z", "last_refresh": "2024-03-11T22:21:19.892553Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-11T22:17:32.323562Z", "last_refresh": "2024-03-11T22:21:19.892970Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-11T22:21:19.893108Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-11T22:17:29.791459Z", "last_refresh": "2024-03-11T22:21:19.893033Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-11T22:24:11.091 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:10 smithi073 ceph-mon[24794]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:11.222 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:10 smithi099 ceph-mon[27157]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:11.710 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: pgmap v218: 65 pgs: 52 active+clean, 2 creating+activating, 7 creating+peering, 4 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:12 smithi099 ceph-mon[27157]: Deploying daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: pgmap v218: 65 pgs: 52 active+clean, 2 creating+activating, 7 creating+peering, 4 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:12.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:12 smithi073 ceph-mon[24794]: Deploying daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:24:12.711 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch ls -f json 2024-03-11T22:24:13.300 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: mds.? [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] up:boot 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: daemon mds.foofs.smithi073.lmopce assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: Cluster is now healthy 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: fsmap foofs:0 1 up:standby 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:creating} 2024-03-11T22:24:13.301 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:13 smithi099 ceph-mon[27157]: daemon mds.foofs.smithi073.lmopce is now active in filesystem foofs as rank 0 2024-03-11T22:24:13.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: mds.? [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] up:boot 2024-03-11T22:24:13.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: daemon mds.foofs.smithi073.lmopce assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-11T22:24:13.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:24:13.332 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T22:24:13.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: Cluster is now healthy 2024-03-11T22:24:13.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: fsmap foofs:0 1 up:standby 2024-03-11T22:24:13.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:24:13.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:creating} 2024-03-11T22:24:13.333 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:13 smithi073 ceph-mon[24794]: daemon mds.foofs.smithi073.lmopce is now active in filesystem foofs as rank 0 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: pgmap v219: 65 pgs: 63 active+clean, 2 creating+activating; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: mds.? [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] up:active 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:14.296 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:14 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: pgmap v219: 65 pgs: 63 active+clean, 2 creating+activating; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 1 op/s 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: mds.? [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] up:active 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:14.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:14 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:14.822 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:24:14.823 INFO:teuthology.orchestra.run.smithi073.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-11T22:17:33.673110Z", "last_refresh": "2024-03-11T22:21:19.892766Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-11T22:17:28.345495Z", "last_refresh": "2024-03-11T22:21:19.892843Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-11T22:17:31.125131Z", "last_refresh": "2024-03-11T22:21:19.892907Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-11T22:24:06.933051Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-11T22:24:06.925899Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-11T22:17:26.767881Z", "last_refresh": "2024-03-11T22:21:19.892696Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi073:172.21.15.73=smithi073", "smithi099:172.21.15.99=smithi099"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-11T22:18:16.830889Z", "last_refresh": "2024-03-11T22:21:19.892553Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-11T22:17:32.323562Z", "last_refresh": "2024-03-11T22:21:19.892970Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-11T22:21:19.893108Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-11T22:17:29.791459Z", "last_refresh": "2024-03-11T22:21:19.893033Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-11T22:24:15.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:15 smithi073 ceph-mon[24794]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:boot 2024-03-11T22:24:15.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:15 smithi073 ceph-mon[24794]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:24:15.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:15 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:24:15.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:15 smithi073 ceph-mon[24794]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:24:15.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:15 smithi073 ceph-mon[24794]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:15.660 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-11T22:24:15.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:15 smithi099 ceph-mon[27157]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:boot 2024-03-11T22:24:15.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:15 smithi099 ceph-mon[27157]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:24:15.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:15 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:24:15.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:15 smithi099 ceph-mon[27157]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:24:15.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:15 smithi099 ceph-mon[27157]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:16.661 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch ls -f json 2024-03-11T22:24:16.686 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:16 smithi073 ceph-mon[24794]: pgmap v220: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 4 op/s 2024-03-11T22:24:16.686 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:16 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:16.686 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:16 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:16.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:16 smithi099 ceph-mon[27157]: pgmap v220: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 4 op/s 2024-03-11T22:24:16.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:16 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:16.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:16 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:18.611 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:18 smithi073 ceph-mon[24794]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-11T22:24:18.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:18 smithi099 ceph-mon[27157]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-11T22:24:20.033 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:24:20.033 INFO:teuthology.orchestra.run.smithi073.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-11T22:17:33.673110Z", "last_refresh": "2024-03-11T22:21:19.892766Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-11T22:17:28.345495Z", "last_refresh": "2024-03-11T22:21:19.892843Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-11T22:17:31.125131Z", "last_refresh": "2024-03-11T22:21:19.892907Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-11T22:24:06.933051Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-11T22:24:06.925899Z", "last_refresh": "2024-03-11T22:24:15.473101Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-11T22:17:26.767881Z", "last_refresh": "2024-03-11T22:21:19.892696Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi073:172.21.15.73=smithi073", "smithi099:172.21.15.99=smithi099"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-11T22:18:16.830889Z", "last_refresh": "2024-03-11T22:21:19.892553Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-11T22:17:32.323562Z", "last_refresh": "2024-03-11T22:21:19.892970Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-11T22:21:19.893108Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-11T22:17:29.791459Z", "last_refresh": "2024-03-11T22:21:19.893033Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-11T22:24:20.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:20 smithi073 ceph-mon[24794]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-11T22:24:20.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:20 smithi099 ceph-mon[27157]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-11T22:24:20.883 INFO:tasks.cephadm:mds.foofs has 1/2 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:21.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:21 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.885 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch ls -f json 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:21 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:22.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:22 smithi099 ceph-mon[27157]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:22.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:22 smithi073 ceph-mon[24794]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:23.804 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:24:23.804 INFO:teuthology.orchestra.run.smithi073.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-11T22:17:33.673110Z", "last_refresh": "2024-03-11T22:24:21.384608Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-11T22:17:28.345495Z", "last_refresh": "2024-03-11T22:24:15.472459Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-11T22:17:31.125131Z", "last_refresh": "2024-03-11T22:24:21.384728Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-11T22:24:06.933051Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-11T22:24:06.925899Z", "last_refresh": "2024-03-11T22:24:15.473101Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-11T22:17:26.767881Z", "last_refresh": "2024-03-11T22:24:15.472595Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi073:172.21.15.73=smithi073", "smithi099:172.21.15.99=smithi099"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-11T22:18:16.830889Z", "last_refresh": "2024-03-11T22:24:15.472664Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-11T22:17:32.323562Z", "last_refresh": "2024-03-11T22:24:15.472725Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-11T22:24:15.472785Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-11T22:17:29.791459Z", "last_refresh": "2024-03-11T22:24:21.384880Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-11T22:24:24.489 INFO:tasks.cephadm:mds.foofs has 2/2 2024-03-11T22:24:24.489 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:24:24.499 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:24.499 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-03-11T22:24:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:24 smithi073 ceph-mon[24794]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:24.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:24 smithi073 ceph-mon[24794]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:24.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:24 smithi099 ceph-mon[27157]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:24.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:24 smithi099 ceph-mon[27157]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:24:26.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:26 smithi099 ceph-mon[27157]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:26.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:26 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:27.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:26 smithi073 ceph-mon[24794]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-11T22:24:27.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:26 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:27.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:27 smithi073 ceph-mon[24794]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:27.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-11T22:24:27.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-11T22:24:27.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:27 smithi073 ceph-mon[24794]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T22:24:27.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:27 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-11T22:24:27.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:27 smithi099 ceph-mon[27157]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:27.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-11T22:24:27.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-11T22:24:27.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:27 smithi099 ceph-mon[27157]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T22:24:27.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:27 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-11T22:24:28.055 INFO:teuthology.orchestra.run.smithi073.stdout:NFS Cluster Created Successfully 2024-03-11T22:24:28.785 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-03-11T22:24:28.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-03-11T22:24:28.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-11T22:24:28.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T22:24:28.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:24:28.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:24:28.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:24:28.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Saving service nfs.foo spec with placement count:2 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Creating key for client.nfs.foo.0.0.smithi073.gkqass 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Rados config object exists: conf-nfs.foo 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Creating key for client.nfs.foo.0.0.smithi073.gkqass-rgw 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: Deploying daemon nfs.foo.0.0.smithi073.gkqass on smithi073 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T22:24:30.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:29 smithi073 ceph-mon[24794]: mgrmap e20: smithi073.pglkvm(active, since 5m), standbys: smithi099.rcoixq 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Saving service nfs.foo spec with placement count:2 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Creating key for client.nfs.foo.0.0.smithi073.gkqass 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Rados config object exists: conf-nfs.foo 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Creating key for client.nfs.foo.0.0.smithi073.gkqass-rgw 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: Deploying daemon nfs.foo.0.0.smithi073.gkqass on smithi073 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T22:24:30.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:29 smithi099 ceph-mon[27157]: mgrmap e20: smithi073.pglkvm(active, since 5m), standbys: smithi099.rcoixq 2024-03-11T22:24:30.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:30 smithi073 ceph-mon[24794]: pgmap v229: 97 pgs: 24 unknown, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:31.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:30 smithi099 ceph-mon[27157]: pgmap v229: 97 pgs: 24 unknown, 73 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:24:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:Invalid command: Unexpected argument '--clusterid' 2024-03-11T22:24:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-03-11T22:24:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:Error EINVAL: invalid command 2024-03-11T22:24:32.252 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:24:32.252 INFO:teuthology.orchestra.run.smithi073.stdout: "bind": "/fake", 2024-03-11T22:24:32.252 INFO:teuthology.orchestra.run.smithi073.stdout: "fs": "foofs", 2024-03-11T22:24:32.252 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-11T22:24:32.252 INFO:teuthology.orchestra.run.smithi073.stdout: "cluster": "foo", 2024-03-11T22:24:32.253 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "RW" 2024-03-11T22:24:32.253 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:24:32.946 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: pgmap v231: 97 pgs: 9 unknown, 88 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:24:33.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-11T22:24:33.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:24:33.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:24:33.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: pgmap v231: 97 pgs: 9 unknown, 88 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:24:33.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:33.897 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:33 smithi073 ceph-mon[24794]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:33.897 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:33 smithi073 ceph-mon[24794]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz 2024-03-11T22:24:33.897 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:33 smithi073 ceph-mon[24794]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-11T22:24:34.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:33 smithi099 ceph-mon[27157]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:34.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:33 smithi099 ceph-mon[27157]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz 2024-03-11T22:24:34.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:33 smithi099 ceph-mon[27157]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-11T22:24:34.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:34 smithi073 ceph-mon[24794]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:24:35.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:34 smithi099 ceph-mon[27157]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:24:35.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:24:35.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:24:35.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:24:35.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:24:35.929 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:35 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:36.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:24:36.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:24:36.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:24:36.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:24:36.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:35 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:37.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:36 smithi099 ceph-mon[27157]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:37.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:36 smithi099 ceph-mon[27157]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-03-11T22:24:37.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:36 smithi099 ceph-mon[27157]: Rados config object exists: conf-nfs.foo 2024-03-11T22:24:37.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:36 smithi099 ceph-mon[27157]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz-rgw 2024-03-11T22:24:37.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:36 smithi099 ceph-mon[27157]: Deploying daemon nfs.foo.1.0.smithi099.ahqxhz on smithi099 2024-03-11T22:24:37.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:36 smithi073 ceph-mon[24794]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:37.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:36 smithi073 ceph-mon[24794]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 383 B/s wr, 0 op/s 2024-03-11T22:24:37.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:36 smithi073 ceph-mon[24794]: Rados config object exists: conf-nfs.foo 2024-03-11T22:24:37.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:36 smithi073 ceph-mon[24794]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz-rgw 2024-03-11T22:24:37.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:36 smithi073 ceph-mon[24794]: Deploying daemon nfs.foo.1.0.smithi099.ahqxhz on smithi099 2024-03-11T22:24:37.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:37 smithi073 ceph-mon[24794]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:37.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:37 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:37.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:37 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:37.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:37 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:37.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:37 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:37.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:37 smithi099 ceph-mon[27157]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:37.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:37 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:37.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:37 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:37.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:37 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:37.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:37 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:39.029 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:38 smithi099 ceph-mon[27157]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T22:24:39.029 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:38 smithi099 ceph-mon[27157]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:39.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:38 smithi073 ceph-mon[24794]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T22:24:39.031 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:38 smithi073 ceph-mon[24794]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:40.035 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:39 smithi099 ceph-mon[27157]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:40.216 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:39 smithi073 ceph-mon[24794]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:41.067 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:40 smithi073 ceph-mon[24794]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-11T22:24:41.067 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:40 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:41.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:40 smithi099 ceph-mon[27157]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-11T22:24:41.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:40 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:42.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:42.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:24:42.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:24:42.138 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:42.807 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.111 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:43.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:43.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:43.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.112 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T22:24:43.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:24:43.525 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-11T22:24:43.534 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:43.534 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-03-11T22:24:43.565 INFO:teuthology.orchestra.run.smithi073.stderr:+ mkdir /mnt/foo 2024-03-11T22:24:43.587 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-03-11T22:24:43.654 INFO:teuthology.orchestra.run.smithi073.stderr:++ hostname 2024-03-11T22:24:43.655 INFO:teuthology.orchestra.run.smithi073.stderr:+ mount -t nfs smithi073:/fake /mnt/foo -o sync 2024-03-11T22:24:44.054 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:43 smithi073 ceph-mon[24794]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:44.054 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:43 smithi073 ceph-mon[24794]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T22:24:44.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-03-11T22:24:44.122 INFO:teuthology.orchestra.run.smithi073.stderr:+ echo test 2024-03-11T22:24:44.156 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-03-11T22:24:44.228 INFO:teuthology.orchestra.run.smithi073.stderr:+ sync 2024-03-11T22:24:44.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:43 smithi099 ceph-mon[27157]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:24:44.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:43 smithi099 ceph-mon[27157]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T22:24:44.470 INFO:teuthology.run_tasks:Running task parallel... 2024-03-11T22:24:44.480 INFO:teuthology.task.parallel:starting parallel... 2024-03-11T22:24:44.480 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-11T22:24:44.481 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-11T22:24:44.481 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:44.481 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-11T22:24:44.481 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-11T22:24:44.482 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-11T22:24:44.482 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T22:24:44.483 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:24:44.483 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-03-11T22:24:44.545 INFO:teuthology.orchestra.run.smithi073.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-11T22:24:44.545 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:24:44.545 INFO:teuthology.orchestra.run.smithi073.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-11T22:24:44.546 INFO:teuthology.orchestra.run.smithi073.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-11T22:24:44.646 INFO:teuthology.orchestra.run.smithi073.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-11T22:24:44.647 INFO:teuthology.orchestra.run.smithi073.stdout:releasing clients 2024-03-11T22:24:45.068 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:44 smithi073 ceph-mon[24794]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-11T22:24:45.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:44 smithi099 ceph-mon[27157]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-11T22:24:45.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 92 21.23 MB/sec warmup 1 sec latency 46.801 ms 2024-03-11T22:24:46.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 190 18.90 MB/sec warmup 2 sec latency 39.442 ms 2024-03-11T22:24:46.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:46 smithi073 ceph-mon[24794]: pgmap v240: 97 pgs: 97 active+clean; 965 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 51 KiB/s wr, 7 op/s 2024-03-11T22:24:47.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:46 smithi099 ceph-mon[27157]: pgmap v240: 97 pgs: 97 active+clean; 965 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 51 KiB/s wr, 7 op/s 2024-03-11T22:24:47.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 303 18.54 MB/sec warmup 3 sec latency 37.956 ms 2024-03-11T22:24:47.768 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-11T22:24:48.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 400 17.72 MB/sec warmup 4 sec latency 37.162 ms 2024-03-11T22:24:48.964 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:48 smithi073 ceph-mon[24794]: pgmap v241: 97 pgs: 97 active+clean; 8.9 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.0 MiB/s wr, 23 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:49.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:48 smithi099 ceph-mon[27157]: pgmap v241: 97 pgs: 97 active+clean; 8.9 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.0 MiB/s wr, 23 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:49.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 508 17.70 MB/sec warmup 5 sec latency 24.081 ms 2024-03-11T22:24:50.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 579 16.46 MB/sec warmup 6 sec latency 86.397 ms 2024-03-11T22:24:50.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:50 smithi073 ceph-mon[24794]: pgmap v242: 97 pgs: 97 active+clean; 34 MiB data, 73 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s wr, 250 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:51.042 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-11T22:24:51.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:50 smithi099 ceph-mon[27157]: pgmap v242: 97 pgs: 97 active+clean; 34 MiB data, 73 MiB used, 715 GiB / 715 GiB avail; 4.3 MiB/s wr, 250 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:51.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 848 15.40 MB/sec warmup 7 sec latency 38.965 ms 2024-03-11T22:24:52.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 1102 14.31 MB/sec warmup 8 sec latency 62.876 ms 2024-03-11T22:24:52.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:52 smithi073 ceph-mon[24794]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 124 MiB used, 715 GiB / 715 GiB avail; 221 B/s rd, 8.8 MiB/s wr, 371 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:53.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:52 smithi099 ceph-mon[27157]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 124 MiB used, 715 GiB / 715 GiB avail; 221 B/s rd, 8.8 MiB/s wr, 371 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:53.646 INFO:teuthology.orchestra.run.smithi073.stdout: 5 1893 14.68 MB/sec warmup 9 sec latency 20.889 ms 2024-03-11T22:24:54.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 2659 15.11 MB/sec warmup 10 sec latency 20.174 ms 2024-03-11T22:24:54.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:54 smithi073 ceph-mon[24794]: pgmap v244: 97 pgs: 97 active+clean; 102 MiB data, 148 MiB used, 715 GiB / 715 GiB avail; 200 B/s rd, 10 MiB/s wr, 395 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:55.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:54 smithi099 ceph-mon[27157]: pgmap v244: 97 pgs: 97 active+clean; 102 MiB data, 148 MiB used, 715 GiB / 715 GiB avail; 200 B/s rd, 10 MiB/s wr, 395 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:55.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 3433 16.64 MB/sec warmup 11 sec latency 44.338 ms 2024-03-11T22:24:55.842 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph mgr module enable nfs --force' 2024-03-11T22:24:56.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 3845 16.59 MB/sec warmup 12 sec latency 77.045 ms 2024-03-11T22:24:56.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:56 smithi073 ceph-mon[24794]: pgmap v245: 97 pgs: 97 active+clean; 107 MiB data, 188 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 9.3 MiB/s wr, 368 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:57.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:56 smithi099 ceph-mon[27157]: pgmap v245: 97 pgs: 97 active+clean; 107 MiB data, 188 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 9.3 MiB/s wr, 368 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:57.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 4358 15.59 MB/sec warmup 13 sec latency 76.902 ms 2024-03-11T22:24:58.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 5026 15.93 MB/sec warmup 14 sec latency 19.116 ms 2024-03-11T22:24:59.028 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:58 smithi073 ceph-mon[24794]: pgmap v246: 97 pgs: 97 active+clean; 119 MiB data, 221 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 11 MiB/s wr, 539 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:59.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:58 smithi099 ceph-mon[27157]: pgmap v246: 97 pgs: 97 active+clean; 119 MiB data, 221 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 11 MiB/s wr, 539 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:24:59.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 5853 15.55 MB/sec warmup 15 sec latency 21.114 ms 2024-03-11T22:24:59.850 INFO:teuthology.orchestra.run.smithi073.stderr:module 'nfs' is already enabled 2024-03-11T22:25:00.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:24:59 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1723577881' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-11T22:25:00.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:24:59 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1723577881' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-11T22:25:00.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 6424 16.18 MB/sec warmup 16 sec latency 31.729 ms 2024-03-11T22:25:01.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:00 smithi073 ceph-mon[24794]: pgmap v247: 97 pgs: 97 active+clean; 133 MiB data, 236 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 13 MiB/s wr, 746 op/s 2024-03-11T22:25:01.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:00 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1723577881' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-11T22:25:01.073 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:00 smithi073 ceph-mon[24794]: mgrmap e21: smithi073.pglkvm(active, since 6m), standbys: smithi099.rcoixq 2024-03-11T22:25:01.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:00 smithi099 ceph-mon[27157]: pgmap v247: 97 pgs: 97 active+clean; 133 MiB data, 236 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 13 MiB/s wr, 746 op/s 2024-03-11T22:25:01.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:00 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1723577881' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-11T22:25:01.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:00 smithi099 ceph-mon[27157]: mgrmap e21: smithi073.pglkvm(active, since 6m), standbys: smithi099.rcoixq 2024-03-11T22:25:01.263 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-11T22:25:01.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 6982 16.39 MB/sec warmup 17 sec latency 29.459 ms 2024-03-11T22:25:02.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 7390 16.36 MB/sec warmup 18 sec latency 32.985 ms 2024-03-11T22:25:03.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:02 smithi073 ceph-mon[24794]: pgmap v248: 97 pgs: 97 active+clean; 139 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 12 MiB/s wr, 759 op/s 2024-03-11T22:25:03.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:02 smithi099 ceph-mon[27157]: pgmap v248: 97 pgs: 97 active+clean; 139 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 12 MiB/s wr, 759 op/s 2024-03-11T22:25:03.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 7806 15.64 MB/sec warmup 19 sec latency 35.127 ms 2024-03-11T22:25:04.336 INFO:teuthology.orchestra.run.smithi073.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:25:04.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 8274 15.31 MB/sec warmup 20 sec latency 21.846 ms 2024-03-11T22:25:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:04 smithi073 ceph-mon[24794]: pgmap v249: 97 pgs: 97 active+clean; 141 MiB data, 281 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 9.5 MiB/s wr, 769 op/s 2024-03-11T22:25:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:25:04.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:04 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:25:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:04 smithi099 ceph-mon[27157]: pgmap v249: 97 pgs: 97 active+clean; 141 MiB data, 281 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 9.5 MiB/s wr, 769 op/s 2024-03-11T22:25:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:25:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:04 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:25:05.590 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-11T22:25:05.591 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:25:05.591 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-11T22:25:05.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 8960 15.38 MB/sec warmup 21 sec latency 33.296 ms 2024-03-11T22:25:06.136 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:05 smithi073 ceph-mon[24794]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:06.136 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:05 smithi073 ceph-mon[24794]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:25:06.136 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:05 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:06.136 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:05 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:06.136 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:05 smithi073 ceph-mon[24794]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:25:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:05 smithi099 ceph-mon[27157]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:05 smithi099 ceph-mon[27157]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:25:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:05 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:05 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:06.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:05 smithi099 ceph-mon[27157]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:25:06.647 INFO:teuthology.orchestra.run.smithi073.stdout: 5 9541 15.12 MB/sec warmup 22 sec latency 38.559 ms 2024-03-11T22:25:06.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:06 smithi073 ceph-mon[24794]: pgmap v250: 97 pgs: 97 active+clean; 142 MiB data, 324 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-03-11T22:25:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:06 smithi099 ceph-mon[27157]: pgmap v250: 97 pgs: 97 active+clean; 142 MiB data, 324 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-03-11T22:25:07.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 10175 15.70 MB/sec warmup 23 sec latency 18.513 ms 2024-03-11T22:25:08.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 10872 16.18 MB/sec warmup 24 sec latency 22.315 ms 2024-03-11T22:25:08.938 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:08 smithi073 ceph-mon[24794]: pgmap v251: 97 pgs: 97 active+clean; 142 MiB data, 373 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 805 op/s 2024-03-11T22:25:09.199 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:25:09.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:08 smithi099 ceph-mon[27157]: pgmap v251: 97 pgs: 97 active+clean; 142 MiB data, 373 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 805 op/s 2024-03-11T22:25:09.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 11259 15.69 MB/sec warmup 25 sec latency 24.732 ms 2024-03-11T22:25:10.234 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:25:10.234 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (5m) 27s ago 6m 25.5M - ba2b418f427c e10c29c255a9 2024-03-11T22:25:10.234 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (6m) 27s ago 6m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (6m) 29s ago 6m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (5m) 27s ago 6m 54.9M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (59s) 27s ago 59s 14.8M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (57s) 29s ago 56s 11.2M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (8m) 27s ago 8m 460M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (6m) 29s ago 6m 412M - 17.2.0 e1d6a67b021e 824e312fca3c 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (8m) 27s ago 8m 47.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (6m) 29s ago 6m 36.8M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (38s) 27s ago 37s 51.1M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (32s) 29s ago 32s 44.5M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 27s ago 6m 18.5M - 1dbe0e931976 46f528d06727 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (5m) 29s ago 5m 19.3M - 1dbe0e931976 fb9193a63c59 2024-03-11T22:25:10.235 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (5m) 27s ago 5m 43.7M 3216M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (4m) 27s ago 4m 46.3M 3216M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (4m) 27s ago 4m 43.3M 3216M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (3m) 27s ago 3m 39.3M 3216M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (3m) 29s ago 3m 41.2M 3984M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (3m) 29s ago 3m 38.9M 3984M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (2m) 29s ago 2m 39.0M 3984M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (2m) 29s ago 2m 42.0M 3984M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:25:10.236 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (5m) 27s ago 5m 62.5M - 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:25:10.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 11788 15.44 MB/sec warmup 26 sec latency 23.237 ms 2024-03-11T22:25:10.858 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:25:10.858 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:25:10.858 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:10.858 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:25:10.859 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:25:10.954 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:10 smithi073 ceph-mon[24794]: pgmap v252: 97 pgs: 97 active+clean; 152 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 856 op/s 2024-03-11T22:25:10.954 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:10 smithi073 ceph-mon[24794]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:10.954 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:10 smithi073 ceph-mon[24794]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:10.954 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:10 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/277782721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:25:11.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:10 smithi099 ceph-mon[27157]: pgmap v252: 97 pgs: 97 active+clean; 152 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 856 op/s 2024-03-11T22:25:11.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:10 smithi099 ceph-mon[27157]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:11.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:10 smithi099 ceph-mon[27157]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:11.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:10 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/277782721' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:25:11.406 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:25:11.406 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T22:25:11.406 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:25:11.406 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:25:11.407 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "", 2024-03-11T22:25:11.407 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf image" 2024-03-11T22:25:11.407 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:25:11.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 12462 15.50 MB/sec warmup 27 sec latency 26.540 ms 2024-03-11T22:25:11.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:11 smithi073 ceph-mon[24794]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:12.061 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:25:12.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:11 smithi099 ceph-mon[27157]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:12.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 13022 15.21 MB/sec warmup 28 sec latency 32.682 ms 2024-03-11T22:25:12.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:12 smithi073 ceph-mon[24794]: pgmap v253: 97 pgs: 97 active+clean; 158 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.5 MiB/s wr, 820 op/s 2024-03-11T22:25:12.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:12 smithi073 ceph-mon[24794]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:12.966 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:12 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/4036974665' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:25:13.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:12 smithi099 ceph-mon[27157]: pgmap v253: 97 pgs: 97 active+clean; 158 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 8.5 MiB/s wr, 820 op/s 2024-03-11T22:25:13.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:12 smithi099 ceph-mon[27157]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:13.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:12 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/4036974665' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:25:13.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 13582 15.57 MB/sec warmup 29 sec latency 21.869 ms 2024-03-11T22:25:14.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 14202 15.88 MB/sec warmup 30 sec latency 34.402 ms 2024-03-11T22:25:14.992 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:14 smithi073 ceph-mon[24794]: pgmap v254: 97 pgs: 97 active+clean; 159 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.4 MiB/s wr, 765 op/s 2024-03-11T22:25:15.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:14 smithi099 ceph-mon[27157]: pgmap v254: 97 pgs: 97 active+clean; 159 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.4 MiB/s wr, 765 op/s 2024-03-11T22:25:15.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 14589 15.70 MB/sec warmup 31 sec latency 24.304 ms 2024-03-11T22:25:16.000 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:15 smithi073 ceph-mon[24794]: pgmap v255: 97 pgs: 97 active+clean; 156 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 7.3 MiB/s wr, 721 op/s 2024-03-11T22:25:16.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:15 smithi099 ceph-mon[27157]: pgmap v255: 97 pgs: 97 active+clean; 156 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 7.3 MiB/s wr, 721 op/s 2024-03-11T22:25:16.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 15262 15.47 MB/sec warmup 32 sec latency 21.961 ms 2024-03-11T22:25:17.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 16021 15.58 MB/sec warmup 33 sec latency 25.117 ms 2024-03-11T22:25:18.263 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:18 smithi073 ceph-mon[24794]: pgmap v256: 97 pgs: 97 active+clean; 163 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-03-11T22:25:18.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:18 smithi099 ceph-mon[27157]: pgmap v256: 97 pgs: 97 active+clean; 163 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-03-11T22:25:18.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 16718 15.51 MB/sec warmup 34 sec latency 25.635 ms 2024-03-11T22:25:19.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 17567 16.07 MB/sec warmup 35 sec latency 15.722 ms 2024-03-11T22:25:20.534 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:20 smithi073 ceph-mon[24794]: pgmap v257: 97 pgs: 97 active+clean; 166 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 803 op/s 2024-03-11T22:25:20.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 18060 16.15 MB/sec warmup 36 sec latency 26.510 ms 2024-03-11T22:25:20.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:20 smithi099 ceph-mon[27157]: pgmap v257: 97 pgs: 97 active+clean; 166 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 803 op/s 2024-03-11T22:25:21.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 18717 15.89 MB/sec warmup 37 sec latency 18.346 ms 2024-03-11T22:25:22.526 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:22 smithi073 ceph-mon[24794]: pgmap v258: 97 pgs: 97 active+clean; 173 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 828 op/s 2024-03-11T22:25:22.648 INFO:teuthology.orchestra.run.smithi073.stdout: 5 19629 16.05 MB/sec warmup 38 sec latency 21.560 ms 2024-03-11T22:25:22.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:22 smithi099 ceph-mon[27157]: pgmap v258: 97 pgs: 97 active+clean; 173 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 828 op/s 2024-03-11T22:25:23.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 20614 16.31 MB/sec warmup 39 sec latency 15.693 ms 2024-03-11T22:25:24.450 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:24 smithi073 ceph-mon[24794]: pgmap v259: 97 pgs: 97 active+clean; 175 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 702 op/s 2024-03-11T22:25:24.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 21473 16.84 MB/sec warmup 40 sec latency 27.970 ms 2024-03-11T22:25:24.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:24 smithi099 ceph-mon[27157]: pgmap v259: 97 pgs: 97 active+clean; 175 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 702 op/s 2024-03-11T22:25:25.649 INFO:teuthology.orchestra.run.smithi073.stdout: 5 21994 16.59 MB/sec warmup 41 sec latency 26.080 ms 2024-03-11T22:25:26.459 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:26 smithi073 ceph-mon[24794]: pgmap v260: 97 pgs: 97 active+clean; 188 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 917 op/s 2024-03-11T22:25:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 22882 16.71 MB/sec warmup 42 sec latency 14.808 ms 2024-03-11T22:25:26.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:26 smithi099 ceph-mon[27157]: pgmap v260: 97 pgs: 97 active+clean; 188 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 917 op/s 2024-03-11T22:25:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 23556 16.52 MB/sec warmup 43 sec latency 31.150 ms 2024-03-11T22:25:28.486 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:28 smithi073 ceph-mon[24794]: pgmap v261: 97 pgs: 97 active+clean; 191 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-03-11T22:25:28.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 24294 16.82 MB/sec warmup 44 sec latency 24.641 ms 2024-03-11T22:25:28.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:28 smithi099 ceph-mon[27157]: pgmap v261: 97 pgs: 97 active+clean; 191 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-03-11T22:25:29.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 24848 16.92 MB/sec warmup 45 sec latency 40.142 ms 2024-03-11T22:25:30.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:30 smithi073 ceph-mon[24794]: pgmap v262: 97 pgs: 97 active+clean; 193 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-03-11T22:25:30.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 25260 16.82 MB/sec warmup 46 sec latency 26.444 ms 2024-03-11T22:25:30.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:30 smithi099 ceph-mon[27157]: pgmap v262: 97 pgs: 97 active+clean; 193 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-03-11T22:25:31.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 25901 16.62 MB/sec warmup 47 sec latency 22.027 ms 2024-03-11T22:25:32.535 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:32 smithi073 ceph-mon[24794]: pgmap v263: 97 pgs: 97 active+clean; 194 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T22:25:32.655 INFO:teuthology.orchestra.run.smithi073.stdout: 5 26646 16.68 MB/sec warmup 48 sec latency 19.745 ms 2024-03-11T22:25:32.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:32 smithi099 ceph-mon[27157]: pgmap v263: 97 pgs: 97 active+clean; 194 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T22:25:33.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 27285 16.57 MB/sec warmup 49 sec latency 34.436 ms 2024-03-11T22:25:34.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:34 smithi073 ceph-mon[24794]: pgmap v264: 97 pgs: 97 active+clean; 203 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 949 op/s 2024-03-11T22:25:34.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 28269 17.06 MB/sec warmup 50 sec latency 72.406 ms 2024-03-11T22:25:34.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:34 smithi099 ceph-mon[27157]: pgmap v264: 97 pgs: 97 active+clean; 203 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 949 op/s 2024-03-11T22:25:35.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 28998 17.07 MB/sec warmup 51 sec latency 13.647 ms 2024-03-11T22:25:36.569 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:36 smithi073 ceph-mon[24794]: pgmap v265: 97 pgs: 97 active+clean; 208 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-03-11T22:25:36.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 29551 16.93 MB/sec warmup 52 sec latency 51.467 ms 2024-03-11T22:25:36.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:36 smithi099 ceph-mon[27157]: pgmap v265: 97 pgs: 97 active+clean; 208 MiB data, 828 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-03-11T22:25:37.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 30259 16.94 MB/sec warmup 53 sec latency 47.089 ms 2024-03-11T22:25:38.599 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:38 smithi073 ceph-mon[24794]: pgmap v266: 97 pgs: 97 active+clean; 211 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 772 op/s 2024-03-11T22:25:38.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 30893 16.83 MB/sec warmup 54 sec latency 67.799 ms 2024-03-11T22:25:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:38 smithi099 ceph-mon[27157]: pgmap v266: 97 pgs: 97 active+clean; 211 MiB data, 872 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 772 op/s 2024-03-11T22:25:39.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 31708 17.12 MB/sec warmup 55 sec latency 62.864 ms 2024-03-11T22:25:40.429 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:40 smithi073 ceph-mon[24794]: pgmap v267: 97 pgs: 97 active+clean; 221 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 997 op/s 2024-03-11T22:25:40.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 32513 17.25 MB/sec warmup 56 sec latency 16.043 ms 2024-03-11T22:25:40.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:40 smithi099 ceph-mon[27157]: pgmap v267: 97 pgs: 97 active+clean; 221 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 997 op/s 2024-03-11T22:25:41.638 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:25:41.638 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:25:41.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 33436 17.28 MB/sec warmup 57 sec latency 50.577 ms 2024-03-11T22:25:41.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:25:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:25:42.616 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:25:42.643 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:42 smithi073 ceph-mon[24794]: pgmap v268: 97 pgs: 97 active+clean; 222 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-03-11T22:25:42.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 34416 17.30 MB/sec warmup 58 sec latency 43.349 ms 2024-03-11T22:25:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:42 smithi099 ceph-mon[27157]: pgmap v268: 97 pgs: 97 active+clean; 222 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 961 op/s 2024-03-11T22:25:43.623 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (6m) 61s ago 7m 25.5M - ba2b418f427c e10c29c255a9 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (7m) 61s ago 7m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (6m) 63s ago 6m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (5m) 61s ago 7m 54.9M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (93s) 61s ago 92s 14.8M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (90s) 63s ago 90s 11.2M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (8m) 61s ago 8m 460M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (6m) 63s ago 6m 412M - 17.2.0 e1d6a67b021e 824e312fca3c 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (8m) 61s ago 8m 47.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (6m) 63s ago 6m 36.8M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (71s) 61s ago 71s 51.1M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:25:43.624 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (66s) 63s ago 65s 44.5M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (7m) 61s ago 7m 18.5M - 1dbe0e931976 46f528d06727 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (6m) 63s ago 6m 19.3M - 1dbe0e931976 fb9193a63c59 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (5m) 61s ago 5m 43.7M 3216M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (5m) 61s ago 5m 46.3M 3216M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (4m) 61s ago 4m 43.3M 3216M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (4m) 61s ago 4m 39.3M 3216M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (4m) 63s ago 4m 41.2M 3984M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (3m) 63s ago 3m 38.9M 3984M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (3m) 63s ago 3m 39.0M 3984M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (2m) 63s ago 2m 42.0M 3984M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:25:43.625 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (6m) 61s ago 6m 62.5M - 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:25:43.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:43 smithi073 ceph-mon[24794]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:43.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 35318 17.60 MB/sec warmup 59 sec latency 41.489 ms 2024-03-11T22:25:43.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:43 smithi099 ceph-mon[27157]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:44.197 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:25:44.198 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:25:44.199 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:25:44.199 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:25:44.199 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-11T22:25:44.199 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:25:44.199 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:25:44.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:44 smithi073 ceph-mon[24794]: pgmap v269: 97 pgs: 97 active+clean; 226 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 988 op/s 2024-03-11T22:25:44.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:44 smithi073 ceph-mon[24794]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:44.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:44 smithi073 ceph-mon[24794]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:44.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:44 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2304852736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:25:44.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 36009 17.69 MB/sec warmup 60 sec latency 15.473 ms 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "", 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf image" 2024-03-11T22:25:44.728 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:25:44.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:44 smithi099 ceph-mon[27157]: pgmap v269: 97 pgs: 97 active+clean; 226 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 988 op/s 2024-03-11T22:25:44.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:44 smithi099 ceph-mon[27157]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:44.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:44 smithi099 ceph-mon[27157]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:44.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:44 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2304852736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:25:45.300 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:25:45.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 36664 17.57 MB/sec warmup 61 sec latency 32.088 ms 2024-03-11T22:25:45.677 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:45 smithi073 ceph-mon[24794]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:45.678 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:45 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1981856241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:25:45.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:45 smithi099 ceph-mon[27157]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:25:45.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:45 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1981856241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:25:46.611 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:46 smithi073 ceph-mon[24794]: pgmap v270: 97 pgs: 97 active+clean; 231 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 955 op/s 2024-03-11T22:25:46.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 37524 17.58 MB/sec warmup 62 sec latency 46.585 ms 2024-03-11T22:25:46.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:46 smithi099 ceph-mon[27157]: pgmap v270: 97 pgs: 97 active+clean; 231 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 955 op/s 2024-03-11T22:25:47.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 38351 17.70 MB/sec warmup 63 sec latency 24.480 ms 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:47.963 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:47 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi073.pglkvm) 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:25:47.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:47 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi073.pglkvm) 2024-03-11T22:25:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 39208 17.99 MB/sec warmup 64 sec latency 21.451 ms 2024-03-11T22:25:48.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:48 smithi073 ceph-mon[24794]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b on smithi099 2024-03-11T22:25:48.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:48 smithi073 ceph-mon[24794]: pgmap v271: 97 pgs: 97 active+clean; 235 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 935 op/s 2024-03-11T22:25:48.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:48 smithi099 ceph-mon[27157]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b on smithi099 2024-03-11T22:25:48.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:48 smithi099 ceph-mon[27157]: pgmap v271: 97 pgs: 97 active+clean; 235 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 935 op/s 2024-03-11T22:25:49.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 39861 17.84 MB/sec warmup 65 sec latency 17.705 ms 2024-03-11T22:25:50.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 40564 17.83 MB/sec warmup 66 sec latency 23.673 ms 2024-03-11T22:25:50.980 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:50 smithi073 ceph-mon[24794]: pgmap v272: 97 pgs: 97 active+clean; 242 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-03-11T22:25:50.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:50 smithi099 ceph-mon[27157]: pgmap v272: 97 pgs: 97 active+clean; 242 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-03-11T22:25:51.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 41399 17.80 MB/sec warmup 67 sec latency 24.484 ms 2024-03-11T22:25:52.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 42330 18.08 MB/sec warmup 68 sec latency 18.321 ms 2024-03-11T22:25:52.741 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:52 smithi073 ceph-mon[24794]: pgmap v273: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 958 op/s 2024-03-11T22:25:52.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:52 smithi099 ceph-mon[27157]: pgmap v273: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 958 op/s 2024-03-11T22:25:53.660 INFO:teuthology.orchestra.run.smithi073.stdout: 5 43190 18.16 MB/sec warmup 69 sec latency 16.354 ms 2024-03-11T22:25:54.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 43991 18.17 MB/sec warmup 70 sec latency 23.330 ms 2024-03-11T22:25:54.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:54 smithi073 ceph-mon[24794]: pgmap v274: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 871 op/s 2024-03-11T22:25:54.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:54 smithi099 ceph-mon[27157]: pgmap v274: 97 pgs: 97 active+clean; 244 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 871 op/s 2024-03-11T22:25:55.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 44818 18.09 MB/sec warmup 71 sec latency 21.973 ms 2024-03-11T22:25:56.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 45508 18.20 MB/sec warmup 72 sec latency 20.473 ms 2024-03-11T22:25:56.785 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:56 smithi073 ceph-mon[24794]: pgmap v275: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-03-11T22:25:57.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:56 smithi099 ceph-mon[27157]: pgmap v275: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-03-11T22:25:57.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 46225 18.34 MB/sec warmup 73 sec latency 25.689 ms 2024-03-11T22:25:58.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 46953 18.25 MB/sec warmup 74 sec latency 40.393 ms 2024-03-11T22:25:58.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:25:58 smithi073 ceph-mon[24794]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 992 op/s 2024-03-11T22:25:58.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:25:58 smithi099 ceph-mon[27157]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 992 op/s 2024-03-11T22:25:59.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 47909 18.32 MB/sec warmup 75 sec latency 15.746 ms 2024-03-11T22:26:00.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 48732 18.29 MB/sec warmup 76 sec latency 16.130 ms 2024-03-11T22:26:00.805 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:00 smithi073 ceph-mon[24794]: pgmap v277: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-03-11T22:26:01.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:00 smithi099 ceph-mon[27157]: pgmap v277: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-03-11T22:26:01.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 49647 18.54 MB/sec warmup 77 sec latency 26.970 ms 2024-03-11T22:26:02.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 50253 18.51 MB/sec warmup 78 sec latency 29.624 ms 2024-03-11T22:26:02.816 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:02 smithi073 ceph-mon[24794]: pgmap v278: 97 pgs: 97 active+clean; 274 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-03-11T22:26:03.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:02 smithi099 ceph-mon[27157]: pgmap v278: 97 pgs: 97 active+clean; 274 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-03-11T22:26:03.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 50911 18.44 MB/sec warmup 79 sec latency 29.205 ms 2024-03-11T22:26:04.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 51772 18.41 MB/sec warmup 80 sec latency 29.150 ms 2024-03-11T22:26:04.824 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:04 smithi073 ceph-mon[24794]: pgmap v279: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-03-11T22:26:05.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:04 smithi099 ceph-mon[27157]: pgmap v279: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-03-11T22:26:05.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 52453 18.43 MB/sec warmup 81 sec latency 25.892 ms 2024-03-11T22:26:06.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 53244 18.61 MB/sec warmup 82 sec latency 25.437 ms 2024-03-11T22:26:06.844 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:06 smithi073 ceph-mon[24794]: pgmap v280: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.26k op/s 2024-03-11T22:26:07.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:06 smithi099 ceph-mon[27157]: pgmap v280: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.26k op/s 2024-03-11T22:26:07.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 53823 18.55 MB/sec warmup 83 sec latency 22.287 ms 2024-03-11T22:26:08.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 54535 18.51 MB/sec warmup 84 sec latency 35.130 ms 2024-03-11T22:26:08.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:08 smithi073 ceph-mon[24794]: pgmap v281: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 999 op/s 2024-03-11T22:26:09.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:08 smithi099 ceph-mon[27157]: pgmap v281: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 999 op/s 2024-03-11T22:26:09.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 55408 18.47 MB/sec warmup 85 sec latency 22.014 ms 2024-03-11T22:26:10.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 56204 18.58 MB/sec warmup 86 sec latency 21.863 ms 2024-03-11T22:26:10.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:10 smithi073 ceph-mon[24794]: pgmap v282: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-03-11T22:26:11.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:10 smithi099 ceph-mon[27157]: pgmap v282: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-03-11T22:26:11.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 56992 18.74 MB/sec warmup 87 sec latency 19.816 ms 2024-03-11T22:26:12.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 57374 18.59 MB/sec warmup 88 sec latency 29.627 ms 2024-03-11T22:26:12.894 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:12 smithi073 ceph-mon[24794]: pgmap v283: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-03-11T22:26:13.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:12 smithi099 ceph-mon[27157]: pgmap v283: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-03-11T22:26:13.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 57880 18.48 MB/sec warmup 89 sec latency 29.060 ms 2024-03-11T22:26:14.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 58562 18.46 MB/sec warmup 90 sec latency 22.409 ms 2024-03-11T22:26:14.907 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:14 smithi073 ceph-mon[24794]: pgmap v284: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 855 op/s 2024-03-11T22:26:15.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:14 smithi099 ceph-mon[27157]: pgmap v284: 97 pgs: 97 active+clean; 295 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 855 op/s 2024-03-11T22:26:15.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 59203 18.35 MB/sec warmup 91 sec latency 35.464 ms 2024-03-11T22:26:15.807 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:26:16.650 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:26:16.650 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (6m) 94s ago 8m 25.5M - ba2b418f427c e10c29c255a9 2024-03-11T22:26:16.650 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (8m) 94s ago 8m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (7m) 96s ago 7m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (6m) 94s ago 7m 54.9M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (2m) 94s ago 2m 14.8M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (2m) 96s ago 2m 11.2M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (9m) 94s ago 9m 460M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (7m) 96s ago 7m 412M - 17.2.0 e1d6a67b021e 824e312fca3c 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (9m) 94s ago 9m 47.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (7m) 96s ago 7m 36.8M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (104s) 94s ago 104s 51.1M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (99s) 96s ago 98s 44.5M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (7m) 94s ago 7m 18.5M - 1dbe0e931976 46f528d06727 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (7m) 96s ago 7m 19.3M - 1dbe0e931976 fb9193a63c59 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (6m) 94s ago 6m 43.7M 3216M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (5m) 94s ago 5m 46.3M 3216M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:26:16.651 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (5m) 94s ago 5m 43.3M 3216M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (5m) 94s ago 5m 39.3M 3216M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (4m) 96s ago 4m 41.2M 3984M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (4m) 96s ago 4m 38.9M 3984M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (3m) 96s ago 3m 39.0M 3984M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (3m) 96s ago 3m 42.0M 3984M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:26:16.652 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (6m) 94s ago 6m 62.5M - 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:26:16.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 59755 18.41 MB/sec warmup 92 sec latency 35.274 ms 2024-03-11T22:26:16.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:16 smithi073 ceph-mon[24794]: pgmap v285: 97 pgs: 97 active+clean; 303 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 9.8 MiB/s wr, 969 op/s 2024-03-11T22:26:16.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:16 smithi073 ceph-mon[24794]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:26:17.144 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:26:17.145 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:26:17.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:16 smithi099 ceph-mon[27157]: pgmap v285: 97 pgs: 97 active+clean; 303 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 9.8 MiB/s wr, 969 op/s 2024-03-11T22:26:17.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:16 smithi099 ceph-mon[27157]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "0/23 daemons upgraded", 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b image on host smithi099" 2024-03-11T22:26:17.574 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:26:17.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 60314 18.46 MB/sec warmup 93 sec latency 36.122 ms 2024-03-11T22:26:17.953 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:17 smithi073 ceph-mon[24794]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:17.953 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:17 smithi073 ceph-mon[24794]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:17.953 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:17 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2907764674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:26:18.084 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:26:18.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:17 smithi099 ceph-mon[27157]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:18.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:17 smithi099 ceph-mon[27157]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:18.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:17 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2907764674' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:26:18.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 60762 18.40 MB/sec warmup 94 sec latency 23.637 ms 2024-03-11T22:26:18.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:18 smithi073 ceph-mon[24794]: pgmap v286: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 852 op/s 2024-03-11T22:26:18.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:18 smithi073 ceph-mon[24794]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:18.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:18 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3187439445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:26:19.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:18 smithi099 ceph-mon[27157]: pgmap v286: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 852 op/s 2024-03-11T22:26:19.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:18 smithi099 ceph-mon[27157]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:19.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:18 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3187439445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:26:19.661 INFO:teuthology.orchestra.run.smithi073.stdout: 5 61413 18.29 MB/sec warmup 95 sec latency 17.048 ms 2024-03-11T22:26:20.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 62375 18.33 MB/sec warmup 96 sec latency 23.466 ms 2024-03-11T22:26:20.974 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:20 smithi073 ceph-mon[24794]: pgmap v287: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 909 op/s 2024-03-11T22:26:21.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:20 smithi099 ceph-mon[27157]: pgmap v287: 97 pgs: 97 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 909 op/s 2024-03-11T22:26:21.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 63052 18.36 MB/sec warmup 97 sec latency 26.762 ms 2024-03-11T22:26:22.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 63493 18.34 MB/sec warmup 98 sec latency 91.538 ms 2024-03-11T22:26:23.019 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:22 smithi073 ceph-mon[24794]: pgmap v288: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.6 MiB/s wr, 889 op/s 2024-03-11T22:26:23.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:22 smithi099 ceph-mon[27157]: pgmap v288: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.6 MiB/s wr, 889 op/s 2024-03-11T22:26:23.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 63893 18.33 MB/sec warmup 99 sec latency 172.898 ms 2024-03-11T22:26:24.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 64190 18.25 MB/sec warmup 100 sec latency 82.865 ms 2024-03-11T22:26:24.921 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:24 smithi073 ceph-mon[24794]: pgmap v289: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.7 MiB/s wr, 747 op/s 2024-03-11T22:26:25.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:24 smithi099 ceph-mon[27157]: pgmap v289: 97 pgs: 97 active+clean; 312 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 8.7 MiB/s wr, 747 op/s 2024-03-11T22:26:25.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 64591 18.09 MB/sec warmup 101 sec latency 63.274 ms 2024-03-11T22:26:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 64918 17.97 MB/sec warmup 102 sec latency 257.392 ms 2024-03-11T22:26:27.033 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:26 smithi073 ceph-mon[24794]: pgmap v290: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.9 MiB/s wr, 959 op/s 2024-03-11T22:26:27.239 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:26 smithi099 ceph-mon[27157]: pgmap v290: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.9 MiB/s wr, 959 op/s 2024-03-11T22:26:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 65457 17.92 MB/sec warmup 103 sec latency 45.530 ms 2024-03-11T22:26:28.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 66141 17.87 MB/sec warmup 104 sec latency 39.213 ms 2024-03-11T22:26:29.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:28 smithi073 ceph-mon[24794]: pgmap v291: 97 pgs: 97 active+clean; 326 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.7 MiB/s wr, 797 op/s 2024-03-11T22:26:29.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:29.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:26:29.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:26:29.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:28 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:29.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:28 smithi099 ceph-mon[27157]: pgmap v291: 97 pgs: 97 active+clean; 326 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.7 MiB/s wr, 797 op/s 2024-03-11T22:26:29.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:29.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:26:29.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:26:29.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:28 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:29.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 66668 17.90 MB/sec warmup 105 sec latency 29.168 ms 2024-03-11T22:26:30.068 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:29 smithi073 ceph-mon[24794]: Upgrade: Updating mgr.smithi099.rcoixq 2024-03-11T22:26:30.068 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:29 smithi073 ceph-mon[24794]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:26:30.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:29 smithi099 ceph-mon[27157]: Upgrade: Updating mgr.smithi099.rcoixq 2024-03-11T22:26:30.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:29 smithi099 ceph-mon[27157]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:26:30.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 67365 17.98 MB/sec warmup 106 sec latency 34.448 ms 2024-03-11T22:26:30.830 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:30 smithi073 ceph-mon[24794]: pgmap v292: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.8 MiB/s wr, 728 op/s 2024-03-11T22:26:31.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:30 smithi099 ceph-mon[27157]: pgmap v292: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.8 MiB/s wr, 728 op/s 2024-03-11T22:26:31.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 67738 17.95 MB/sec warmup 107 sec latency 33.229 ms 2024-03-11T22:26:32.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 68245 17.81 MB/sec warmup 108 sec latency 24.314 ms 2024-03-11T22:26:32.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:32 smithi073 ceph-mon[24794]: pgmap v293: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 825 op/s 2024-03-11T22:26:32.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:32.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:32.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:32.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:32 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:26:32.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:32 smithi099 ceph-mon[27157]: pgmap v293: 97 pgs: 97 active+clean; 327 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 825 op/s 2024-03-11T22:26:32.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:32.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:32.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:32 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:26:33.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 68586 17.73 MB/sec warmup 109 sec latency 27.837 ms 2024-03-11T22:26:34.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 69027 17.68 MB/sec warmup 110 sec latency 66.966 ms 2024-03-11T22:26:34.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:34 smithi073 ceph-mon[24794]: pgmap v294: 97 pgs: 97 active+clean; 325 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 737 op/s 2024-03-11T22:26:34.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:34 smithi099 ceph-mon[27157]: pgmap v294: 97 pgs: 97 active+clean; 325 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 737 op/s 2024-03-11T22:26:35.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 69632 17.60 MB/sec warmup 111 sec latency 53.432 ms 2024-03-11T22:26:36.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 70137 17.61 MB/sec warmup 112 sec latency 24.448 ms 2024-03-11T22:26:36.895 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:36 smithi073 ceph-mon[24794]: pgmap v295: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.7 MiB/s wr, 905 op/s 2024-03-11T22:26:36.895 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:36 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:37.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:36 smithi099 ceph-mon[27157]: pgmap v295: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.7 MiB/s wr, 905 op/s 2024-03-11T22:26:37.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:36 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:37.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 70666 17.64 MB/sec warmup 113 sec latency 28.943 ms 2024-03-11T22:26:38.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:38 smithi099 ceph-mon[27157]: pgmap v296: 97 pgs: 97 active+clean; 324 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-11T22:26:38.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:38 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:38.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:38 smithi073 ceph-mon[24794]: pgmap v296: 97 pgs: 97 active+clean; 324 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-11T22:26:38.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:38 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:38.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 71324 17.69 MB/sec warmup 114 sec latency 25.405 ms 2024-03-11T22:26:39.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 71950 17.59 MB/sec warmup 115 sec latency 23.024 ms 2024-03-11T22:26:40.417 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:40 smithi073 ceph-mon[24794]: pgmap v297: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.1 MiB/s wr, 803 op/s 2024-03-11T22:26:40.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 72779 17.62 MB/sec warmup 116 sec latency 26.637 ms 2024-03-11T22:26:40.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:40 smithi099 ceph-mon[27157]: pgmap v297: 97 pgs: 97 active+clean; 317 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 7.1 MiB/s wr, 803 op/s 2024-03-11T22:26:41.430 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:41.430 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:41.430 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:41.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr fail", "who": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:26:41.431 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:41 smithi073 ceph-mon[24794]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T22:26:41.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 73484 17.58 MB/sec warmup 117 sec latency 27.222 ms 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr fail", "who": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:26:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:41 smithi099 ceph-mon[27157]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: pgmap v298: 97 pgs: 97 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 877 op/s 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi073.pglkvm) 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Failing over to other MGR 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Active manager daemon smithi099.rcoixq restarted 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Activating manager daemon smithi099.rcoixq 2024-03-11T22:26:42.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "mgr fail", "who": "smithi073.pglkvm"}]': finished 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: mgrmap e22: smithi099.rcoixq(active, starting, since 0.758084s), standbys: smithi073.pglkvm 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:26:42.447 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: Manager daemon smithi099.rcoixq is now available 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:42.448 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:42.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:42.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:42.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:42.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:42.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:42 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: pgmap v298: 97 pgs: 97 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 877 op/s 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi073.pglkvm) 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Failing over to other MGR 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Active manager daemon smithi099.rcoixq restarted 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Activating manager daemon smithi099.rcoixq 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:26:42.473 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.14204 172.21.15.73:0/1156017512' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "mgr fail", "who": "smithi073.pglkvm"}]': finished 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: mgrmap e22: smithi099.rcoixq(active, starting, since 0.758084s), standbys: smithi073.pglkvm 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:26:42.474 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: Manager daemon smithi099.rcoixq is now available 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:42.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:42 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.rcoixq/trash_purge_schedule"}]: dispatch 2024-03-11T22:26:42.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 74186 17.70 MB/sec warmup 118 sec latency 27.996 ms 2024-03-11T22:26:43.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 74714 17.72 MB/sec warmup 119 sec latency 38.124 ms 2024-03-11T22:26:43.973 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: Deploying cephadm binary to smithi099 2024-03-11T22:26:43.973 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: Deploying cephadm binary to smithi073 2024-03-11T22:26:43.973 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:43.974 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:43.974 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: mgrmap e23: smithi099.rcoixq(active, since 1.75914s), standbys: smithi073.pglkvm 2024-03-11T22:26:43.974 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:43 smithi073 ceph-mon[24794]: pgmap v3: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: Deploying cephadm binary to smithi099 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: Deploying cephadm binary to smithi073 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: mgrmap e23: smithi099.rcoixq(active, since 1.75914s), standbys: smithi073.pglkvm 2024-03-11T22:26:44.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:43 smithi099 ceph-mon[27157]: pgmap v3: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: [11/Mar/2024:22:26:43] ENGINE Bus STARTING 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: [11/Mar/2024:22:26:43] ENGINE Serving on https://172.21.15.99:7150 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: [11/Mar/2024:22:26:43] ENGINE Serving on http://172.21.15.99:8765 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: [11/Mar/2024:22:26:43] ENGINE Bus STARTED 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: pgmap v4: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:45.193 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:45 smithi073 ceph-mon[24794]: mgrmap e24: smithi099.rcoixq(active, since 2s), standbys: smithi073.pglkvm 2024-03-11T22:26:45.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: [11/Mar/2024:22:26:43] ENGINE Bus STARTING 2024-03-11T22:26:45.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: [11/Mar/2024:22:26:43] ENGINE Serving on https://172.21.15.99:7150 2024-03-11T22:26:45.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: [11/Mar/2024:22:26:43] ENGINE Serving on http://172.21.15.99:8765 2024-03-11T22:26:45.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: [11/Mar/2024:22:26:43] ENGINE Bus STARTED 2024-03-11T22:26:45.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: pgmap v4: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:45.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:45 smithi099 ceph-mon[27157]: mgrmap e24: smithi099.rcoixq(active, since 2s), standbys: smithi073.pglkvm 2024-03-11T22:26:45.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 75515 6.47 MB/sec execute 1 sec latency 30.910 ms 2024-03-11T22:26:46.663 INFO:teuthology.orchestra.run.smithi073.stdout: 5 76132 11.42 MB/sec execute 2 sec latency 29.721 ms 2024-03-11T22:26:47.242 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:47 smithi073 ceph-mon[24794]: pgmap v5: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:47.242 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:47 smithi073 ceph-mon[24794]: mgrmap e25: smithi099.rcoixq(active, since 4s), standbys: smithi073.pglkvm 2024-03-11T22:26:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:47 smithi099 ceph-mon[27157]: pgmap v5: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:47 smithi099 ceph-mon[27157]: mgrmap e25: smithi099.rcoixq(active, since 4s), standbys: smithi073.pglkvm 2024-03-11T22:26:47.662 INFO:teuthology.orchestra.run.smithi073.stdout: 5 76810 10.69 MB/sec execute 3 sec latency 29.053 ms 2024-03-11T22:26:48.650 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:26:48.663 INFO:teuthology.orchestra.run.smithi073.stdout: 5 77267 12.40 MB/sec execute 4 sec latency 41.780 ms 2024-03-11T22:26:49.263 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:49 smithi073 ceph-mon[24794]: pgmap v6: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:49.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:49 smithi099 ceph-mon[27157]: pgmap v6: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-11T22:26:49.663 INFO:teuthology.orchestra.run.smithi073.stdout: 5 77672 13.03 MB/sec execute 5 sec latency 45.309 ms 2024-03-11T22:26:49.752 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:26:49.752 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (7m) 2m ago 8m 25.5M - ba2b418f427c e10c29c255a9 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (8m) 2m ago 8m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (7m) 13s ago 7m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (6m) 2m ago 8m 54.9M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (2m) 2m ago 2m 14.8M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (2m) 13s ago 2m 11.7M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (17s) 13s ago 7m 47.1M - 19.0.0-1607-g94a53970 a6477a474d71 edc9a06dac71 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (9m) 2m ago 9m 47.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (7m) 13s ago 7m 37.8M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (2m) 2m ago 2m 51.1M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (2m) 13s ago 2m 70.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (8m) 2m ago 8m 18.5M - 1dbe0e931976 46f528d06727 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (7m) 13s ago 7m 16.1M - 1dbe0e931976 fb9193a63c59 2024-03-11T22:26:49.753 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (7m) 2m ago 7m 43.7M 3216M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (6m) 2m ago 6m 46.3M 3216M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (6m) 2m ago 6m 43.3M 3216M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (5m) 2m ago 5m 39.3M 3216M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (5m) 13s ago 5m 246M 3984M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (4m) 13s ago 4m 281M 3984M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (4m) 13s ago 4m 249M 3984M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (4m) 13s ago 4m 358M 3984M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:26:49.754 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (7m) 2m ago 7m 62.5M - 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:26:50.274 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:50 smithi073 ceph-mon[24794]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:50.274 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:50 smithi073 ceph-mon[24794]: from='client.24477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:26:50.452 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:26:50.453 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:26:50.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:50 smithi099 ceph-mon[27157]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:50.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:50 smithi099 ceph-mon[27157]: from='client.24477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:50.664 INFO:teuthology.orchestra.run.smithi073.stdout: 5 78109 13.82 MB/sec execute 6 sec latency 31.064 ms 2024-03-11T22:26:51.010 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "", 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:26:51.011 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:26:51.298 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:51 smithi073 ceph-mon[24794]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:51.298 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:51 smithi073 ceph-mon[24794]: pgmap v7: 97 pgs: 97 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.8 MiB/s wr, 540 op/s 2024-03-11T22:26:51.298 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:51 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/865587552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:26:51.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:51 smithi099 ceph-mon[27157]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:51.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:51 smithi099 ceph-mon[27157]: pgmap v7: 97 pgs: 97 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.8 MiB/s wr, 540 op/s 2024-03-11T22:26:51.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:51 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/865587552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:26:51.610 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:26:51.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 78489 13.53 MB/sec execute 7 sec latency 41.206 ms 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: Standby manager daemon smithi073.pglkvm restarted 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/crt"}]: dispatch 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:26:52.308 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/key"}]: dispatch 2024-03-11T22:26:52.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:26:52.309 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:52 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/4193343466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:26:52.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:26:52.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: Standby manager daemon smithi073.pglkvm restarted 2024-03-11T22:26:52.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:26:52.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/crt"}]: dispatch 2024-03-11T22:26:52.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:26:52.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/key"}]: dispatch 2024-03-11T22:26:52.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/4269863201' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:26:52.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:52 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/4193343466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:26:52.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 78919 12.37 MB/sec execute 8 sec latency 64.998 ms 2024-03-11T22:26:53.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:53 smithi073 ceph-mon[24794]: pgmap v8: 97 pgs: 97 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.7 MiB/s wr, 420 op/s 2024-03-11T22:26:53.329 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:53 smithi073 ceph-mon[24794]: mgrmap e26: smithi099.rcoixq(active, since 10s), standbys: smithi073.pglkvm 2024-03-11T22:26:53.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:53 smithi099 ceph-mon[27157]: pgmap v8: 97 pgs: 97 active+clean; 313 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.7 MiB/s wr, 420 op/s 2024-03-11T22:26:53.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:53 smithi099 ceph-mon[27157]: mgrmap e26: smithi099.rcoixq(active, since 10s), standbys: smithi073.pglkvm 2024-03-11T22:26:53.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 79321 11.95 MB/sec execute 9 sec latency 51.841 ms 2024-03-11T22:26:54.592 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:54 smithi073 ceph-mon[24794]: pgmap v9: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-03-11T22:26:54.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 79902 12.30 MB/sec execute 10 sec latency 46.612 ms 2024-03-11T22:26:54.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:54 smithi099 ceph-mon[27157]: pgmap v9: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-03-11T22:26:55.598 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:55 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:55.598 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:55 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:55.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 80564 12.15 MB/sec execute 11 sec latency 25.290 ms 2024-03-11T22:26:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:55 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:55 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:56.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 81040 12.88 MB/sec execute 12 sec latency 56.325 ms 2024-03-11T22:26:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:56 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:56 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:56.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:56 smithi073 ceph-mon[24794]: pgmap v10: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 574 op/s 2024-03-11T22:26:56.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:56 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:56.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:56 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:56.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:56 smithi099 ceph-mon[27157]: pgmap v10: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 574 op/s 2024-03-11T22:26:57.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 81601 13.64 MB/sec execute 13 sec latency 68.433 ms 2024-03-11T22:26:58.139 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:58 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:58.139 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:58 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:58.139 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:58 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:58.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:58 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:58.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:58 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:58.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:58 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:58.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 81979 13.56 MB/sec execute 14 sec latency 33.677 ms 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: pgmap v11: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 574 op/s 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.151 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.152 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:26:59 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: pgmap v11: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 574 op/s 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:26:59.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:26:59 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:26:59.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 82409 12.91 MB/sec execute 15 sec latency 44.358 ms 2024-03-11T22:27:00.165 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi073 to 1936M 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:27:00.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:00 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi073 to 1936M 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:27:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:00 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:27:00.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 82807 12.64 MB/sec execute 16 sec latency 28.459 ms 2024-03-11T22:27:01.181 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Adjusting osd_memory_target on smithi099 to 2704M 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: pgmap v12: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 9.1 MiB/s wr, 882 op/s 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:01 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Adjusting osd_memory_target on smithi099 to 2704M 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:27:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: pgmap v12: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 9.1 MiB/s wr, 882 op/s 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:01 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:01.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 83361 12.79 MB/sec execute 17 sec latency 33.453 ms 2024-03-11T22:27:02.207 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:27:02.207 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:27:02.207 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-03-11T22:27:02.208 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:02 smithi073 ceph-mon[24794]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-03-11T22:27:02.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:27:02.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:27:02.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:27:02.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:02.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: Reconfiguring alertmanager.smithi073 (dependencies changed)... 2024-03-11T22:27:02.471 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:02 smithi099 ceph-mon[27157]: Reconfiguring daemon alertmanager.smithi073 on smithi073 2024-03-11T22:27:02.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 83944 12.45 MB/sec execute 18 sec latency 33.920 ms 2024-03-11T22:27:03.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:03 smithi073 ceph-mon[24794]: pgmap v13: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.3 MiB/s wr, 568 op/s 2024-03-11T22:27:03.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 84455 12.88 MB/sec execute 19 sec latency 48.069 ms 2024-03-11T22:27:03.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:03 smithi099 ceph-mon[27157]: pgmap v13: 97 pgs: 97 active+clean; 318 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.3 MiB/s wr, 568 op/s 2024-03-11T22:27:04.485 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:04 smithi073 ceph-mon[24794]: pgmap v14: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 862 op/s 2024-03-11T22:27:04.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 85198 13.70 MB/sec execute 20 sec latency 77.912 ms 2024-03-11T22:27:04.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:04 smithi099 ceph-mon[27157]: pgmap v14: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 862 op/s 2024-03-11T22:27:05.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 85711 13.65 MB/sec execute 21 sec latency 30.741 ms 2024-03-11T22:27:06.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 86207 13.43 MB/sec execute 22 sec latency 29.288 ms 2024-03-11T22:27:07.440 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:07 smithi073 ceph-mon[24794]: pgmap v15: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-03-11T22:27:07.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 86818 13.47 MB/sec execute 23 sec latency 36.855 ms 2024-03-11T22:27:07.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:07 smithi099 ceph-mon[27157]: pgmap v15: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-03-11T22:27:08.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 87637 13.57 MB/sec execute 24 sec latency 32.197 ms 2024-03-11T22:27:09.271 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:09 smithi073 ceph-mon[24794]: pgmap v16: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-03-11T22:27:09.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 88228 13.98 MB/sec execute 25 sec latency 49.136 ms 2024-03-11T22:27:09.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:09 smithi099 ceph-mon[27157]: pgmap v16: 97 pgs: 97 active+clean; 315 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-03-11T22:27:10.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 88770 14.23 MB/sec execute 26 sec latency 46.967 ms 2024-03-11T22:27:11.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:11 smithi073 ceph-mon[24794]: pgmap v17: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 955 op/s 2024-03-11T22:27:11.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 89137 14.14 MB/sec execute 27 sec latency 48.648 ms 2024-03-11T22:27:11.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:11 smithi099 ceph-mon[27157]: pgmap v17: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.5 MiB/s wr, 955 op/s 2024-03-11T22:27:12.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 89574 13.77 MB/sec execute 28 sec latency 32.733 ms 2024-03-11T22:27:13.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:13 smithi073 ceph-mon[24794]: pgmap v18: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.2 MiB/s wr, 648 op/s 2024-03-11T22:27:13.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:13 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:13.665 INFO:teuthology.orchestra.run.smithi073.stdout: 5 90065 13.72 MB/sec execute 29 sec latency 22.942 ms 2024-03-11T22:27:13.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:13 smithi099 ceph-mon[27157]: pgmap v18: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.2 MiB/s wr, 648 op/s 2024-03-11T22:27:13.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:13 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:14.502 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:14 smithi073 ceph-mon[24794]: pgmap v19: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 959 op/s 2024-03-11T22:27:14.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 90580 13.68 MB/sec execute 30 sec latency 47.803 ms 2024-03-11T22:27:14.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:14 smithi099 ceph-mon[27157]: pgmap v19: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 959 op/s 2024-03-11T22:27:15.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 91128 13.54 MB/sec execute 31 sec latency 40.594 ms 2024-03-11T22:27:16.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 91589 13.69 MB/sec execute 32 sec latency 32.993 ms 2024-03-11T22:27:17.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:17 smithi073 ceph-mon[24794]: pgmap v20: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-03-11T22:27:17.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 92192 13.96 MB/sec execute 33 sec latency 21.989 ms 2024-03-11T22:27:17.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:17 smithi099 ceph-mon[27157]: pgmap v20: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-03-11T22:27:18.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 92666 14.04 MB/sec execute 34 sec latency 42.531 ms 2024-03-11T22:27:19.320 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:19 smithi073 ceph-mon[24794]: pgmap v21: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-03-11T22:27:19.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 93151 13.83 MB/sec execute 35 sec latency 21.413 ms 2024-03-11T22:27:19.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:19 smithi099 ceph-mon[27157]: pgmap v21: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-03-11T22:27:20.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 93612 13.63 MB/sec execute 36 sec latency 33.212 ms 2024-03-11T22:27:21.337 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: pgmap v22: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-03-11T22:27:21.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.338 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:27:21.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 94233 13.71 MB/sec execute 37 sec latency 29.798 ms 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: pgmap v22: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:27:22.205 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:27:22.341 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:22 smithi073 ceph-mon[24794]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-03-11T22:27:22.341 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:22 smithi073 ceph-mon[24794]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:27:22.341 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:27:22.341 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:22 smithi073 ceph-mon[24794]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-03-11T22:27:22.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 94931 13.80 MB/sec execute 38 sec latency 24.427 ms 2024-03-11T22:27:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:22 smithi099 ceph-mon[27157]: Reconfiguring grafana.smithi073 (dependencies changed)... 2024-03-11T22:27:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:22 smithi099 ceph-mon[27157]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:27:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:27:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:22 smithi099 ceph-mon[27157]: Reconfiguring daemon grafana.smithi073 on smithi073 2024-03-11T22:27:23.160 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:27:23.161 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 starting - - - - 2024-03-11T22:27:23.161 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (9m) 26s ago 9m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:27:23.161 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (8m) 28s ago 8m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:27:23.161 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (7m) 26s ago 8m 55.6M - 8.3.5 dad864ee21e9 b315ae0ffb16 2024-03-11T22:27:23.161 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (3m) 26s ago 3m 37.0M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (3m) 28s ago 3m 12.1M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (10m) 26s ago 10m 334M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (51s) 28s ago 8m 490M - 19.0.0-1607-g94a53970 a6477a474d71 edc9a06dac71 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (10m) 26s ago 10m 53.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (8m) 28s ago 8m 40.7M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (2m) 26s ago 2m 242M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (2m) 28s ago 2m 73.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (8m) 26s ago 8m 19.7M - 1.3.1 1dbe0e931976 46f528d06727 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (8m) 28s ago 8m 16.2M - 1.3.1 1dbe0e931976 fb9193a63c59 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (7m) 26s ago 7m 312M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (7m) 26s ago 7m 246M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:27:23.163 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (6m) 26s ago 6m 200M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (6m) 26s ago 6m 273M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (5m) 28s ago 5m 280M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (5m) 28s ago 5m 314M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (5m) 28s ago 5m 267M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (4m) 28s ago 4m 388M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:27:23.164 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (7m) 26s ago 7m 65.3M - 2.33.4 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:27:23.348 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:23 smithi073 ceph-mon[24794]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:23.348 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:23 smithi073 ceph-mon[24794]: pgmap v23: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.8 MiB/s wr, 614 op/s 2024-03-11T22:27:23.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 95412 14.02 MB/sec execute 39 sec latency 32.040 ms 2024-03-11T22:27:23.719 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:27:23.719 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:27:23.719 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:27:23.719 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:23.719 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:27:23.720 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:27:23.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:23 smithi099 ceph-mon[27157]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:23.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:23 smithi099 ceph-mon[27157]: pgmap v23: 97 pgs: 97 active+clean; 323 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.8 MiB/s wr, 614 op/s 2024-03-11T22:27:24.181 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:27:24.181 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:27:24.181 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:27:24.181 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:27:24.182 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:27:24.182 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T22:27:24.182 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "", 2024-03-11T22:27:24.182 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:27:24.182 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:27:24.366 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:24 smithi073 ceph-mon[24794]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:24.366 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:24 smithi073 ceph-mon[24794]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:24.366 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:24 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1512689377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:27:24.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 95838 13.94 MB/sec execute 40 sec latency 34.349 ms 2024-03-11T22:27:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:24 smithi099 ceph-mon[27157]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:24 smithi099 ceph-mon[27157]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:24 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1512689377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:27:24.751 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:27:25.386 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:25 smithi073 ceph-mon[24794]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:25.386 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:25 smithi073 ceph-mon[24794]: pgmap v24: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 947 op/s 2024-03-11T22:27:25.386 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:25 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/328231435' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:27:25.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 96183 13.95 MB/sec execute 41 sec latency 42.634 ms 2024-03-11T22:27:25.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:25 smithi099 ceph-mon[27157]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:25.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:25 smithi099 ceph-mon[27157]: pgmap v24: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 947 op/s 2024-03-11T22:27:25.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:25 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/328231435' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:27:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: 5 96444 13.71 MB/sec execute 42 sec latency 41.497 ms 2024-03-11T22:27:27.456 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:27 smithi073 ceph-mon[24794]: pgmap v25: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-03-11T22:27:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 96811 13.50 MB/sec execute 43 sec latency 34.491 ms 2024-03-11T22:27:27.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:27 smithi099 ceph-mon[27157]: pgmap v25: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-03-11T22:27:28.461 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:28 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:28.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 97480 13.59 MB/sec execute 44 sec latency 33.349 ms 2024-03-11T22:27:28.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:28 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:29.376 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:29 smithi073 ceph-mon[24794]: pgmap v26: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-03-11T22:27:29.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 98211 13.57 MB/sec execute 45 sec latency 33.602 ms 2024-03-11T22:27:29.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:29 smithi099 ceph-mon[27157]: pgmap v26: 97 pgs: 97 active+clean; 320 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-03-11T22:27:30.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 98634 13.62 MB/sec execute 46 sec latency 34.801 ms 2024-03-11T22:27:31.490 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:31 smithi073 ceph-mon[24794]: pgmap v27: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 885 op/s 2024-03-11T22:27:31.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:31 smithi099 ceph-mon[27157]: pgmap v27: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 885 op/s 2024-03-11T22:27:31.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 99169 13.80 MB/sec execute 47 sec latency 31.471 ms 2024-03-11T22:27:32.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 99600 13.84 MB/sec execute 48 sec latency 34.878 ms 2024-03-11T22:27:33.530 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:33 smithi073 ceph-mon[24794]: pgmap v28: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-03-11T22:27:33.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 99853 13.67 MB/sec execute 49 sec latency 45.368 ms 2024-03-11T22:27:33.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:33 smithi099 ceph-mon[27157]: pgmap v28: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-03-11T22:27:34.537 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:34 smithi073 ceph-mon[24794]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 882 op/s 2024-03-11T22:27:34.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 100177 13.50 MB/sec execute 50 sec latency 43.369 ms 2024-03-11T22:27:34.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:34 smithi099 ceph-mon[27157]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 882 op/s 2024-03-11T22:27:35.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 100636 13.33 MB/sec execute 51 sec latency 66.540 ms 2024-03-11T22:27:36.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 101074 13.36 MB/sec execute 52 sec latency 35.252 ms 2024-03-11T22:27:37.436 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:37 smithi073 ceph-mon[24794]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-11T22:27:37.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 101588 13.31 MB/sec execute 53 sec latency 90.853 ms 2024-03-11T22:27:37.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:37 smithi099 ceph-mon[27157]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-11T22:27:38.668 INFO:teuthology.orchestra.run.smithi073.stdout: 5 101991 13.24 MB/sec execute 54 sec latency 35.725 ms 2024-03-11T22:27:39.467 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:39 smithi073 ceph-mon[24794]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-11T22:27:39.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 102227 13.16 MB/sec execute 55 sec latency 48.328 ms 2024-03-11T22:27:39.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:39 smithi099 ceph-mon[27157]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-11T22:27:40.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 102587 13.21 MB/sec execute 56 sec latency 33.168 ms 2024-03-11T22:27:41.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:41 smithi073 ceph-mon[24794]: pgmap v32: 97 pgs: 97 active+clean; 316 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 803 op/s 2024-03-11T22:27:41.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 103033 13.22 MB/sec execute 57 sec latency 45.260 ms 2024-03-11T22:27:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:41 smithi099 ceph-mon[27157]: pgmap v32: 97 pgs: 97 active+clean; 316 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 803 op/s 2024-03-11T22:27:42.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 103372 13.20 MB/sec execute 58 sec latency 88.889 ms 2024-03-11T22:27:43.537 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:43 smithi073 ceph-mon[24794]: pgmap v33: 97 pgs: 97 active+clean; 316 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-03-11T22:27:43.537 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:43 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:43.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 103759 13.06 MB/sec execute 59 sec latency 42.006 ms 2024-03-11T22:27:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:43 smithi099 ceph-mon[27157]: pgmap v33: 97 pgs: 97 active+clean; 316 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-03-11T22:27:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:43 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:44.547 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:44 smithi073 ceph-mon[24794]: pgmap v34: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-11T22:27:44.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 104185 12.93 MB/sec execute 60 sec latency 37.069 ms 2024-03-11T22:27:44.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:44 smithi099 ceph-mon[27157]: pgmap v34: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-11T22:27:45.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 104536 12.86 MB/sec execute 61 sec latency 52.415 ms 2024-03-11T22:27:46.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 105066 12.91 MB/sec execute 62 sec latency 35.316 ms 2024-03-11T22:27:47.354 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:47 smithi073 ceph-mon[24794]: pgmap v35: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-03-11T22:27:47.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 105638 12.93 MB/sec execute 63 sec latency 26.743 ms 2024-03-11T22:27:47.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:47 smithi099 ceph-mon[27157]: pgmap v35: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-03-11T22:27:48.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 106016 12.95 MB/sec execute 64 sec latency 51.542 ms 2024-03-11T22:27:49.369 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:49 smithi073 ceph-mon[24794]: pgmap v36: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.8 MiB/s wr, 474 op/s 2024-03-11T22:27:49.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:49 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:49.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:49 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:49.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:49 smithi073 ceph-mon[24794]: Reconfiguring node-exporter.smithi073 (dependencies changed)... 2024-03-11T22:27:49.370 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:49 smithi073 ceph-mon[24794]: Reconfiguring daemon node-exporter.smithi073 on smithi073 2024-03-11T22:27:49.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 106487 13.00 MB/sec execute 65 sec latency 29.257 ms 2024-03-11T22:27:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:49 smithi099 ceph-mon[27157]: pgmap v36: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.8 MiB/s wr, 474 op/s 2024-03-11T22:27:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:49 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:49 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:49 smithi099 ceph-mon[27157]: Reconfiguring node-exporter.smithi073 (dependencies changed)... 2024-03-11T22:27:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:49 smithi099 ceph-mon[27157]: Reconfiguring daemon node-exporter.smithi073 on smithi073 2024-03-11T22:27:50.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 106847 13.02 MB/sec execute 66 sec latency 33.419 ms 2024-03-11T22:27:51.405 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:51 smithi073 ceph-mon[24794]: pgmap v37: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 743 op/s 2024-03-11T22:27:51.405 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:51 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:51.405 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:51 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:51.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 107115 12.92 MB/sec execute 67 sec latency 53.251 ms 2024-03-11T22:27:51.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:51 smithi099 ceph-mon[27157]: pgmap v37: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 743 op/s 2024-03-11T22:27:51.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:51 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:51.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:51 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:27:52.422 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:52 smithi073 ceph-mon[24794]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-03-11T22:27:52.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 107440 12.75 MB/sec execute 68 sec latency 48.376 ms 2024-03-11T22:27:52.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:52 smithi099 ceph-mon[27157]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-03-11T22:27:53.435 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:53 smithi073 ceph-mon[24794]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-03-11T22:27:53.436 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:53 smithi073 ceph-mon[24794]: pgmap v38: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 490 op/s 2024-03-11T22:27:53.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 107794 12.69 MB/sec execute 69 sec latency 53.194 ms 2024-03-11T22:27:53.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:53 smithi099 ceph-mon[27157]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-03-11T22:27:53.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:53 smithi099 ceph-mon[27157]: pgmap v38: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 490 op/s 2024-03-11T22:27:54.461 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:54 smithi073 ceph-mon[24794]: pgmap v39: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.5 MiB/s wr, 713 op/s 2024-03-11T22:27:54.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 108247 12.70 MB/sec execute 70 sec latency 44.658 ms 2024-03-11T22:27:54.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:54 smithi099 ceph-mon[27157]: pgmap v39: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.5 MiB/s wr, 713 op/s 2024-03-11T22:27:55.340 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:27:55.467 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:55 smithi073 ceph-mon[24794]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:55.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 108832 12.66 MB/sec execute 71 sec latency 43.195 ms 2024-03-11T22:27:55.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:55 smithi099 ceph-mon[27157]: from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:56.461 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:27:56.461 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 starting - - - - 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (9m) 59s ago 9m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (9m) 61s ago 9m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 starting - - - - 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (3m) 59s ago 3m 37.0M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (3m) 61s ago 3m 12.1M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:27:56.462 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (10m) 59s ago 10m 334M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (84s) 61s ago 8m 490M - 19.0.0-1607-g94a53970 a6477a474d71 edc9a06dac71 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (10m) 59s ago 11m 53.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (8m) 61s ago 8m 40.7M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (3m) 59s ago 3m 242M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (3m) 61s ago 3m 73.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 starting - - - - 2024-03-11T22:27:56.463 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (8m) 61s ago 8m 16.2M - 1.3.1 1dbe0e931976 fb9193a63c59 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (8m) 59s ago 8m 312M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (7m) 59s ago 7m 246M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (7m) 59s ago 7m 200M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (6m) 59s ago 6m 273M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (6m) 61s ago 6m 280M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:27:56.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (5m) 61s ago 5m 314M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:27:56.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (5m) 61s ago 5m 267M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:27:56.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (5m) 61s ago 5m 388M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:27:56.465 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (8m) 59s ago 8m 65.3M - 2.33.4 514e6a882f6e 0cd4ff6fa7d4 2024-03-11T22:27:56.482 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:56 smithi073 ceph-mon[24794]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:56.483 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:56 smithi073 ceph-mon[24794]: pgmap v40: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 496 op/s 2024-03-11T22:27:56.669 INFO:teuthology.orchestra.run.smithi073.stdout: 5 109213 12.65 MB/sec execute 72 sec latency 63.380 ms 2024-03-11T22:27:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:56 smithi099 ceph-mon[27157]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:56 smithi099 ceph-mon[27157]: pgmap v40: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 496 op/s 2024-03-11T22:27:57.149 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:27:57.149 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:27:57.150 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-11T22:27:57.151 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:27:57.151 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:27:57.151 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:27:57.492 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:57 smithi073 ceph-mon[24794]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:57.492 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:57 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3281387293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:27:57.493 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:57 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:57.649 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:27:57.649 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:27:57.649 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:27:57.649 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:27:57.650 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:27:57.650 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T22:27:57.650 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "", 2024-03-11T22:27:57.650 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:27:57.650 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:27:57.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 109679 12.74 MB/sec execute 73 sec latency 35.088 ms 2024-03-11T22:27:57.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:57 smithi099 ceph-mon[27157]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:57.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:57 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3281387293' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:27:57.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:57 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:58.247 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:27:58.500 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:58 smithi073 ceph-mon[24794]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:58.501 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:58 smithi073 ceph-mon[24794]: pgmap v41: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-03-11T22:27:58.501 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:27:58 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3856865368' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:27:58.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 110304 12.87 MB/sec execute 74 sec latency 45.815 ms 2024-03-11T22:27:58.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:58 smithi099 ceph-mon[27157]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:27:58.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:58 smithi099 ceph-mon[27157]: pgmap v41: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-03-11T22:27:58.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:27:58 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3856865368' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:27:59.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 110677 12.81 MB/sec execute 75 sec latency 54.460 ms 2024-03-11T22:28:00.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 111154 12.70 MB/sec execute 76 sec latency 33.186 ms 2024-03-11T22:28:01.303 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:01 smithi073 ceph-mon[24794]: pgmap v42: 97 pgs: 97 active+clean; 326 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 776 op/s 2024-03-11T22:28:01.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 111510 12.66 MB/sec execute 77 sec latency 41.998 ms 2024-03-11T22:28:01.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:01 smithi099 ceph-mon[27157]: pgmap v42: 97 pgs: 97 active+clean; 326 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 776 op/s 2024-03-11T22:28:02.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 111990 12.69 MB/sec execute 78 sec latency 53.817 ms 2024-03-11T22:28:03.441 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:03 smithi073 ceph-mon[24794]: pgmap v43: 97 pgs: 97 active+clean; 326 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.5 MiB/s wr, 506 op/s 2024-03-11T22:28:03.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 112410 12.60 MB/sec execute 79 sec latency 51.923 ms 2024-03-11T22:28:03.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:03 smithi099 ceph-mon[27157]: pgmap v43: 97 pgs: 97 active+clean; 326 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.5 MiB/s wr, 506 op/s 2024-03-11T22:28:04.467 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:04 smithi073 ceph-mon[24794]: pgmap v44: 97 pgs: 97 active+clean; 322 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 778 op/s 2024-03-11T22:28:04.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 112772 12.59 MB/sec execute 80 sec latency 33.862 ms 2024-03-11T22:28:04.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:04 smithi099 ceph-mon[27157]: pgmap v44: 97 pgs: 97 active+clean; 322 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 778 op/s 2024-03-11T22:28:05.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 113061 12.56 MB/sec execute 81 sec latency 53.426 ms 2024-03-11T22:28:06.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 113428 12.59 MB/sec execute 82 sec latency 62.275 ms 2024-03-11T22:28:07.323 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:07 smithi073 ceph-mon[24794]: pgmap v45: 97 pgs: 97 active+clean; 322 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s 2024-03-11T22:28:07.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 113810 12.59 MB/sec execute 83 sec latency 30.938 ms 2024-03-11T22:28:07.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:07 smithi099 ceph-mon[27157]: pgmap v45: 97 pgs: 97 active+clean; 322 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.4 MiB/s wr, 554 op/s 2024-03-11T22:28:08.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 114118 12.54 MB/sec execute 84 sec latency 41.685 ms 2024-03-11T22:28:09.355 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:09 smithi073 ceph-mon[24794]: pgmap v46: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 601 op/s 2024-03-11T22:28:09.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 114521 12.45 MB/sec execute 85 sec latency 36.124 ms 2024-03-11T22:28:09.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:09 smithi099 ceph-mon[27157]: pgmap v46: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 601 op/s 2024-03-11T22:28:10.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 114928 12.41 MB/sec execute 86 sec latency 43.109 ms 2024-03-11T22:28:11.404 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:11 smithi073 ceph-mon[24794]: pgmap v47: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 717 op/s 2024-03-11T22:28:11.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 115407 12.43 MB/sec execute 87 sec latency 33.572 ms 2024-03-11T22:28:11.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:11 smithi099 ceph-mon[27157]: pgmap v47: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 717 op/s 2024-03-11T22:28:12.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 115954 12.39 MB/sec execute 88 sec latency 32.186 ms 2024-03-11T22:28:13.429 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:13 smithi073 ceph-mon[24794]: pgmap v48: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-03-11T22:28:13.430 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:13 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:13.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 116518 12.47 MB/sec execute 89 sec latency 26.855 ms 2024-03-11T22:28:13.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:13 smithi099 ceph-mon[27157]: pgmap v48: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-03-11T22:28:13.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:13 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:14.440 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:14 smithi073 ceph-mon[24794]: pgmap v49: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 761 op/s 2024-03-11T22:28:14.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 117154 12.61 MB/sec execute 90 sec latency 36.677 ms 2024-03-11T22:28:14.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:14 smithi099 ceph-mon[27157]: pgmap v49: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 761 op/s 2024-03-11T22:28:15.445 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:15 smithi073 ceph-mon[24794]: mgrmap e27: smithi099.rcoixq(active, since 92s), standbys: smithi073.pglkvm 2024-03-11T22:28:15.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 117774 12.66 MB/sec execute 91 sec latency 39.194 ms 2024-03-11T22:28:15.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:15 smithi099 ceph-mon[27157]: mgrmap e27: smithi099.rcoixq(active, since 92s), standbys: smithi073.pglkvm 2024-03-11T22:28:16.449 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:16 smithi073 ceph-mon[24794]: pgmap v50: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-03-11T22:28:16.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 118443 12.64 MB/sec execute 92 sec latency 24.397 ms 2024-03-11T22:28:16.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:16 smithi099 ceph-mon[27157]: pgmap v50: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-03-11T22:28:17.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 119091 12.69 MB/sec execute 93 sec latency 55.106 ms 2024-03-11T22:28:18.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 119677 12.67 MB/sec execute 94 sec latency 43.316 ms 2024-03-11T22:28:19.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:19 smithi073 ceph-mon[24794]: pgmap v51: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-03-11T22:28:19.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:19 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:19.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:19 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:19.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:19 smithi073 ceph-mon[24794]: Reconfiguring node-exporter.smithi099 (dependencies changed)... 2024-03-11T22:28:19.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:19 smithi073 ceph-mon[24794]: Reconfiguring daemon node-exporter.smithi099 on smithi099 2024-03-11T22:28:19.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:19 smithi099 ceph-mon[27157]: pgmap v51: 97 pgs: 97 active+clean; 321 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-03-11T22:28:19.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:19 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:19.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:19 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:19.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:19 smithi099 ceph-mon[27157]: Reconfiguring node-exporter.smithi099 (dependencies changed)... 2024-03-11T22:28:19.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:19 smithi099 ceph-mon[27157]: Reconfiguring daemon node-exporter.smithi099 on smithi099 2024-03-11T22:28:19.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 120050 12.68 MB/sec execute 95 sec latency 41.732 ms 2024-03-11T22:28:20.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 120460 12.74 MB/sec execute 96 sec latency 42.662 ms 2024-03-11T22:28:21.267 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.6 MiB/s wr, 809 op/s 2024-03-11T22:28:21.267 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:21 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:21.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 120806 12.70 MB/sec execute 97 sec latency 41.701 ms 2024-03-11T22:28:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 8.6 MiB/s wr, 809 op/s 2024-03-11T22:28:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:21 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:22.281 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:22 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:22.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 121246 12.70 MB/sec execute 98 sec latency 33.594 ms 2024-03-11T22:28:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:28:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi073.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:28:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:28:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi073.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:28:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:28:22.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi073.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:28:22.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:22.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:22 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:23.291 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:23 smithi073 ceph-mon[24794]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 648 op/s 2024-03-11T22:28:23.671 INFO:teuthology.orchestra.run.smithi073.stdout: 5 121783 12.64 MB/sec execute 99 sec latency 31.764 ms 2024-03-11T22:28:23.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:23 smithi099 ceph-mon[27157]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 648 op/s 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: Upgrade: Updating mgr.smithi073.pglkvm 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: Deploying daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:28:24.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:24 smithi073 ceph-mon[24794]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.2 MiB/s wr, 910 op/s 2024-03-11T22:28:24.670 INFO:teuthology.orchestra.run.smithi073.stdout: 5 122273 12.65 MB/sec execute 100 sec latency 24.369 ms 2024-03-11T22:28:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: Upgrade: Updating mgr.smithi073.pglkvm 2024-03-11T22:28:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:28:24.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:28:24.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:28:24.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:24.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: Deploying daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:28:24.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:24 smithi099 ceph-mon[27157]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.2 MiB/s wr, 910 op/s 2024-03-11T22:28:25.671 INFO:teuthology.orchestra.run.smithi073.stdout: 5 122953 12.68 MB/sec execute 101 sec latency 27.973 ms 2024-03-11T22:28:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: 5 123425 12.70 MB/sec execute 102 sec latency 45.134 ms 2024-03-11T22:28:27.322 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:27 smithi073 ceph-mon[24794]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 629 op/s 2024-03-11T22:28:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 123870 12.74 MB/sec execute 103 sec latency 37.510 ms 2024-03-11T22:28:27.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:27 smithi099 ceph-mon[27157]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 629 op/s 2024-03-11T22:28:28.334 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:28 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:28.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 124319 12.76 MB/sec execute 104 sec latency 47.558 ms 2024-03-11T22:28:28.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:28 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:28.756 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:28:29.345 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:29 smithi073 ceph-mon[24794]: pgmap v56: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 641 op/s 2024-03-11T22:28:29.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 124708 12.75 MB/sec execute 105 sec latency 59.376 ms 2024-03-11T22:28:29.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:29 smithi099 ceph-mon[27157]: pgmap v56: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 641 op/s 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 starting - - - - 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (10m) 92s ago 10m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (9m) 95s ago 9m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 starting - - - - 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (4m) 92s ago 4m 37.0M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:28:29.801 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (4m) 95s ago 4m 12.1M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:9283 running (11m) 92s ago 11m 334M - 17.2.0 e1d6a67b021e 61ac1e45cd72 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (117s) 95s ago 9m 490M - 19.0.0-1607-g94a53970 a6477a474d71 edc9a06dac71 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (11m) 92s ago 11m 53.2M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (9m) 95s ago 9m 40.7M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (3m) 92s ago 3m 242M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (3m) 95s ago 3m 73.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 starting - - - - 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 starting - - - - 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (8m) 92s ago 8m 312M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (8m) 92s ago 8m 246M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (7m) 92s ago 7m 200M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (7m) 92s ago 7m 273M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:28:29.802 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (6m) 95s ago 6m 280M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:28:29.803 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (6m) 95s ago 6m 314M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:28:29.803 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (6m) 95s ago 6m 267M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:28:29.803 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (5m) 95s ago 5m 388M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:28:29.803 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 starting - - - - 2024-03-11T22:28:30.352 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:30 smithi073 ceph-mon[24794]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:30.353 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:30 smithi073 ceph-mon[24794]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:28:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:28:30.503 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:28:30.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 125084 12.69 MB/sec execute 106 sec latency 59.528 ms 2024-03-11T22:28:30.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:30 smithi099 ceph-mon[27157]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:30.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:30 smithi099 ceph-mon[27157]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "1/23 daemons upgraded", 2024-03-11T22:28:31.085 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading mgr daemons", 2024-03-11T22:28:31.086 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:28:31.086 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:28:31.361 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:31 smithi073 ceph-mon[24794]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:31.361 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:31 smithi073 ceph-mon[24794]: pgmap v57: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.8 MiB/s wr, 926 op/s 2024-03-11T22:28:31.361 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:31 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/2375145151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:28:31.609 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:31 smithi099 ceph-mon[27157]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:31 smithi099 ceph-mon[27157]: pgmap v57: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 9.8 MiB/s wr, 926 op/s 2024-03-11T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:31 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/2375145151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:28:31.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 125498 12.65 MB/sec execute 107 sec latency 36.812 ms 2024-03-11T22:28:32.382 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:32 smithi073 ceph-mon[24794]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:32.382 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:32 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3869481233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:28:32.382 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:32 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:32.382 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:32 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:32.382 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:32 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:32.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 125827 12.60 MB/sec execute 108 sec latency 74.770 ms 2024-03-11T22:28:32.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:32 smithi099 ceph-mon[27157]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:28:32.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:32 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3869481233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:28:32.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:32 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:32.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:32 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:32.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:32 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:33.416 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:33 smithi073 ceph-mon[24794]: pgmap v58: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-03-11T22:28:33.416 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:33 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:33.416 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:33 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:33.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 126313 12.60 MB/sec execute 109 sec latency 69.076 ms 2024-03-11T22:28:33.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:33 smithi099 ceph-mon[27157]: pgmap v58: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-03-11T22:28:33.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:33 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:33.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:33 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:34.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 126915 12.62 MB/sec execute 110 sec latency 30.692 ms 2024-03-11T22:28:34.678 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:34 smithi073 ceph-mon[24794]: pgmap v59: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 821 op/s 2024-03-11T22:28:34.679 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:34 smithi099 ceph-mon[27157]: pgmap v59: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 821 op/s 2024-03-11T22:28:35.672 INFO:teuthology.orchestra.run.smithi073.stdout: 5 127367 12.68 MB/sec execute 111 sec latency 34.254 ms 2024-03-11T22:28:36.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 127789 12.69 MB/sec execute 112 sec latency 37.705 ms 2024-03-11T22:28:37.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:37 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:37.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:37 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:37.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:37 smithi073 ceph-mon[24794]: pgmap v60: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 558 op/s 2024-03-11T22:28:37.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:37 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:37.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:37 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:37.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:37 smithi099 ceph-mon[27157]: pgmap v60: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 558 op/s 2024-03-11T22:28:37.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 128081 12.66 MB/sec execute 113 sec latency 46.967 ms 2024-03-11T22:28:38.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 128376 12.62 MB/sec execute 114 sec latency 48.591 ms 2024-03-11T22:28:39.290 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:39 smithi073 ceph-mon[24794]: pgmap v61: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 660 op/s 2024-03-11T22:28:39.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 128659 12.53 MB/sec execute 115 sec latency 64.854 ms 2024-03-11T22:28:39.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:39 smithi099 ceph-mon[27157]: pgmap v61: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 660 op/s 2024-03-11T22:28:40.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 128922 12.45 MB/sec execute 116 sec latency 49.326 ms 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 8.2 MiB/s wr, 800 op/s 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: Standby manager daemon smithi073.pglkvm restarted 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/crt"}]: dispatch 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/key"}]: dispatch 2024-03-11T22:28:41.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:41 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:28:41.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 129128 12.39 MB/sec execute 117 sec latency 63.355 ms 2024-03-11T22:28:41.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 8.2 MiB/s wr, 800 op/s 2024-03-11T22:28:41.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: Standby manager daemon smithi073.pglkvm restarted 2024-03-11T22:28:41.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: Standby manager daemon smithi073.pglkvm started 2024-03-11T22:28:41.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/crt"}]: dispatch 2024-03-11T22:28:41.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:28:41.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi073.pglkvm/key"}]: dispatch 2024-03-11T22:28:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:41 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:28:42.356 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:42 smithi073 ceph-mon[24794]: mgrmap e28: smithi099.rcoixq(active, since 119s), standbys: smithi073.pglkvm 2024-03-11T22:28:42.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 129503 12.39 MB/sec execute 118 sec latency 52.095 ms 2024-03-11T22:28:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:42 smithi099 ceph-mon[27157]: mgrmap e28: smithi099.rcoixq(active, since 119s), standbys: smithi073.pglkvm 2024-03-11T22:28:43.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:43 smithi073 ceph-mon[24794]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 508 op/s 2024-03-11T22:28:43.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:43 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:43.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:43 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:43.371 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:43 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:43.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 129932 12.36 MB/sec execute 119 sec latency 43.220 ms 2024-03-11T22:28:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:43 smithi099 ceph-mon[27157]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.3 MiB/s wr, 508 op/s 2024-03-11T22:28:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:43 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:43 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:43.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:43 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:44.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 130363 12.35 MB/sec execute 120 sec latency 52.109 ms 2024-03-11T22:28:44.941 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:44 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:44.941 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:44 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:44.941 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:44 smithi073 ceph-mon[24794]: pgmap v64: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 701 op/s 2024-03-11T22:28:45.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:44 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:45.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:44 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:45.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:44 smithi099 ceph-mon[27157]: pgmap v64: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 701 op/s 2024-03-11T22:28:45.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 130692 12.33 MB/sec execute 121 sec latency 31.272 ms 2024-03-11T22:28:46.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 131138 12.35 MB/sec execute 122 sec latency 53.552 ms 2024-03-11T22:28:47.467 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:47 smithi073 ceph-mon[24794]: pgmap v65: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-03-11T22:28:47.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 131602 12.40 MB/sec execute 123 sec latency 33.039 ms 2024-03-11T22:28:47.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:47 smithi099 ceph-mon[27157]: pgmap v65: 97 pgs: 97 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-03-11T22:28:48.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 131959 12.38 MB/sec execute 124 sec latency 61.220 ms 2024-03-11T22:28:49.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:49 smithi073 ceph-mon[24794]: pgmap v66: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.7 MiB/s wr, 452 op/s 2024-03-11T22:28:49.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 132262 12.30 MB/sec execute 125 sec latency 51.003 ms 2024-03-11T22:28:49.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:49 smithi099 ceph-mon[27157]: pgmap v66: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.7 MiB/s wr, 452 op/s 2024-03-11T22:28:50.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 132552 12.25 MB/sec execute 126 sec latency 44.207 ms 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: Failing over to other MGR 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:28:51.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:50 smithi073 ceph-mon[24794]: pgmap v68: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 525 op/s 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: Upgrade: Need to upgrade myself (mgr.smithi099.rcoixq) 2024-03-11T22:28:51.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: Failing over to other MGR 2024-03-11T22:28:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 172.21.15.99:0/1011253943' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:28:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:28:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:28:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:50 smithi099 ceph-mon[27157]: pgmap v68: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.3 MiB/s wr, 525 op/s 2024-03-11T22:28:51.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 132737 12.18 MB/sec execute 127 sec latency 98.913 ms 2024-03-11T22:28:52.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd='[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]': finished 2024-03-11T22:28:52.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: mgrmap e29: smithi073.pglkvm(active, starting, since 0.859677s) 2024-03-11T22:28:52.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:28:52.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:28:52.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:28:52.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:28:52.060 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:28:52.061 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:28:52.061 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:52.061 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:52.061 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:52.061 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:28:52.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:51 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.24445 ' entity='mgr.smithi099.rcoixq' cmd='[{"prefix": "mgr fail", "who": "smithi099.rcoixq"}]': finished 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: mgrmap e29: smithi073.pglkvm(active, starting, since 0.859677s) 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:28:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:28:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:51 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:28:52.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 133044 12.18 MB/sec execute 128 sec latency 50.415 ms 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: mgrmap e30: smithi073.pglkvm(active, since 1.86774s) 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: [11/Mar/2024:22:28:52] ENGINE Bus STARTING 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: [11/Mar/2024:22:28:52] ENGINE Serving on http://172.21.15.73:8765 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: [11/Mar/2024:22:28:52] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:28:53.092 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:52 smithi073 ceph-mon[24794]: [11/Mar/2024:22:28:52] ENGINE Bus STARTED 2024-03-11T22:28:53.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: mgrmap e30: smithi073.pglkvm(active, since 1.86774s) 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: [11/Mar/2024:22:28:52] ENGINE Bus STARTING 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: [11/Mar/2024:22:28:52] ENGINE Serving on http://172.21.15.73:8765 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: [11/Mar/2024:22:28:52] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:28:53.250 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:52 smithi099 ceph-mon[27157]: [11/Mar/2024:22:28:52] ENGINE Bus STARTED 2024-03-11T22:28:53.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 133399 12.12 MB/sec execute 129 sec latency 56.817 ms 2024-03-11T22:28:54.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:54 smithi099 ceph-mon[27157]: pgmap v4: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:54.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:54 smithi099 ceph-mon[27157]: mgrmap e31: smithi073.pglkvm(active, since 3s) 2024-03-11T22:28:54.638 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:54 smithi073 ceph-mon[24794]: pgmap v4: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:54.638 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:54 smithi073 ceph-mon[24794]: mgrmap e31: smithi073.pglkvm(active, since 3s) 2024-03-11T22:28:54.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 133759 12.10 MB/sec execute 130 sec latency 46.927 ms 2024-03-11T22:28:55.673 INFO:teuthology.orchestra.run.smithi073.stdout: 5 134020 12.07 MB/sec execute 131 sec latency 64.014 ms 2024-03-11T22:28:56.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:56 smithi073 ceph-mon[24794]: pgmap v5: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:56.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:56 smithi099 ceph-mon[27157]: pgmap v5: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:56.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 134386 12.07 MB/sec execute 132 sec latency 45.546 ms 2024-03-11T22:28:57.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 134764 12.08 MB/sec execute 133 sec latency 38.051 ms 2024-03-11T22:28:57.760 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:57 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:57.760 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:57 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:57.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:57 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:57.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:57 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:58.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 135121 12.09 MB/sec execute 134 sec latency 40.226 ms 2024-03-11T22:28:58.771 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:58 smithi073 ceph-mon[24794]: pgmap v6: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:58.771 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:58 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:58.771 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:28:58 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:58.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:58 smithi099 ceph-mon[27157]: pgmap v6: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:28:58.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:58 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:58.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:28:58 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:28:59.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 135402 12.06 MB/sec execute 135 sec latency 50.069 ms 2024-03-11T22:29:00.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 135612 11.99 MB/sec execute 136 sec latency 80.391 ms 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 3.5 MiB/s wr, 346 op/s 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:00.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:00.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:00 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:00.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 3.5 MiB/s wr, 346 op/s 2024-03-11T22:29:00.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/3714209489' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:00 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:01.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 135863 11.93 MB/sec execute 137 sec latency 80.552 ms 2024-03-11T22:29:01.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:01 smithi073 ceph-mon[24794]: mgrmap e32: smithi073.pglkvm(active, since 10s), standbys: smithi099.rcoixq 2024-03-11T22:29:01.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:01 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:01.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:01 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:01.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:01 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:01.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:01 smithi099 ceph-mon[27157]: mgrmap e32: smithi073.pglkvm(active, since 10s), standbys: smithi099.rcoixq 2024-03-11T22:29:01.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:01 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:01.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:01 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:01.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:01 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:02.067 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:29:02.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 136098 11.88 MB/sec execute 138 sec latency 65.241 ms 2024-03-11T22:29:02.871 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:29:02.871 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (103s) 2s ago 10m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (10m) 2s ago 10m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (10m) 6s ago 10m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (75s) 2s ago 10m 43.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (4m) 2s ago 4m 37.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (4m) 6s ago 4m 12.9M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (31s) 2s ago 12m 476M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283 running (2m) 6s ago 10m 102M - 19.0.0-1607-g94a53970 a6477a474d71 edc9a06dac71 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (12m) 2s ago 12m 57.3M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (10m) 6s ago 10m 41.8M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (4m) 2s ago 4m 271M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (4m) 6s ago 4m 83.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (72s) 2s ago 10m 8539k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (42s) 6s ago 9m 10.7M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (9m) 2s ago 9m 476M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:29:02.872 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (8m) 2s ago 8m 408M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (8m) 2s ago 8m 294M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (7m) 2s ago 7m 441M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (7m) 6s ago 7m 460M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (7m) 6s ago 7m 501M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (6m) 6s ago 6m 420M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (6m) 6s ago 6m 329M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:29:02.873 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (45s) 2s ago 9m 51.0M - 2.33.4 514e6a882f6e d7d966034932 2024-03-11T22:29:02.903 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:02 smithi073 ceph-mon[24794]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 269 op/s 2024-03-11T22:29:02.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:02 smithi099 ceph-mon[27157]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 269 op/s 2024-03-11T22:29:03.397 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:29:03.397 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:29:03.397 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-11T22:29:03.398 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:29:03.399 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:29:03.399 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:29:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 136405 11.84 MB/sec execute 139 sec latency 74.220 ms 2024-03-11T22:29:03.811 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:29:03.812 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:29:03.812 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:29:03.812 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:29:03.812 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:29:03.812 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr" 2024-03-11T22:29:03.813 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:29:03.813 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "2/23 daemons upgraded", 2024-03-11T22:29:03.813 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "", 2024-03-11T22:29:03.813 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:29:03.813 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:29:03.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:03 smithi073 ceph-mon[24794]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:03.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:03 smithi073 ceph-mon[24794]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:03.967 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:03 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/4147666592' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:03.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:03 smithi099 ceph-mon[27157]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:03 smithi099 ceph-mon[27157]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:03 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/4147666592' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:04.376 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:29:04.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 136774 11.84 MB/sec execute 140 sec latency 38.567 ms 2024-03-11T22:29:04.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:04 smithi073 ceph-mon[24794]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:04.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:04 smithi073 ceph-mon[24794]: pgmap v9: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-03-11T22:29:04.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:04 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/3754378213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:29:04.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:04 smithi099 ceph-mon[27157]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:04.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:04 smithi099 ceph-mon[27157]: pgmap v9: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-03-11T22:29:04.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:04 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/3754378213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:29:05.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 137225 11.82 MB/sec execute 141 sec latency 35.784 ms 2024-03-11T22:29:05.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:05 smithi073 ceph-mon[24794]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:05.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:05 smithi099 ceph-mon[27157]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:06.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 137554 11.80 MB/sec execute 142 sec latency 35.231 ms 2024-03-11T22:29:06.800 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:06 smithi073 ceph-mon[24794]: pgmap v10: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 22 KiB/s rd, 3.9 MiB/s wr, 384 op/s 2024-03-11T22:29:06.800 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:06 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:06.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:06 smithi099 ceph-mon[27157]: pgmap v10: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 22 KiB/s rd, 3.9 MiB/s wr, 384 op/s 2024-03-11T22:29:06.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:06 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:07.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 137935 11.81 MB/sec execute 143 sec latency 39.592 ms 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: pgmap v11: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 22 KiB/s rd, 3.9 MiB/s wr, 384 op/s 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:29:08.071 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:29:08.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:29:08.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:29:08.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:08 smithi073 ceph-mon[24794]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: pgmap v11: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 22 KiB/s rd, 3.9 MiB/s wr, 384 op/s 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:29:08.378 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:08 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:29:08.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 138257 11.80 MB/sec execute 144 sec latency 46.456 ms 2024-03-11T22:29:09.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:29:09.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:29:09.346 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-03-11T22:29:09.347 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:09 smithi073 ceph-mon[24794]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-03-11T22:29:09.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:29:09.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: Reconfiguring prometheus.smithi073 (dependencies changed)... 2024-03-11T22:29:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:09 smithi099 ceph-mon[27157]: Reconfiguring daemon prometheus.smithi073 on smithi073 2024-03-11T22:29:09.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 138639 11.82 MB/sec execute 145 sec latency 45.039 ms 2024-03-11T22:29:10.623 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:10 smithi073 ceph-mon[24794]: pgmap v12: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 597 op/s 2024-03-11T22:29:10.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 138973 11.81 MB/sec execute 146 sec latency 57.779 ms 2024-03-11T22:29:10.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:10 smithi099 ceph-mon[27157]: pgmap v12: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 597 op/s 2024-03-11T22:29:11.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 139300 11.75 MB/sec execute 147 sec latency 39.526 ms 2024-03-11T22:29:12.475 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:12 smithi073 ceph-mon[24794]: pgmap v13: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 397 op/s 2024-03-11T22:29:12.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 139556 11.70 MB/sec execute 148 sec latency 49.457 ms 2024-03-11T22:29:12.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:12 smithi099 ceph-mon[27157]: pgmap v13: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 397 op/s 2024-03-11T22:29:13.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 140163 11.74 MB/sec execute 149 sec latency 36.659 ms 2024-03-11T22:29:14.661 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:14 smithi073 ceph-mon[24794]: pgmap v14: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-03-11T22:29:14.674 INFO:teuthology.orchestra.run.smithi073.stdout: 5 140784 11.75 MB/sec execute 150 sec latency 26.904 ms 2024-03-11T22:29:14.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:14 smithi099 ceph-mon[27157]: pgmap v14: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.1 MiB/s wr, 601 op/s 2024-03-11T22:29:15.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 141365 11.79 MB/sec execute 151 sec latency 34.167 ms 2024-03-11T22:29:16.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 141940 11.84 MB/sec execute 152 sec latency 39.618 ms 2024-03-11T22:29:16.675 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:16 smithi073 ceph-mon[24794]: pgmap v15: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-03-11T22:29:16.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:16 smithi099 ceph-mon[27157]: pgmap v15: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-03-11T22:29:17.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 142433 11.89 MB/sec execute 153 sec latency 25.852 ms 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: pgmap v16: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:29:18.437 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:18 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:18.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 142908 11.84 MB/sec execute 154 sec latency 53.422 ms 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: pgmap v16: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 416 op/s 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:29:18.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:18 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:19.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:19 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:29:19.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:19 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:29:19.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:19 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:19.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 143255 11.82 MB/sec execute 155 sec latency 41.506 ms 2024-03-11T22:29:19.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:19 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:29:19.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:19 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:29:19.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:19 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:20.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:20 smithi073 ceph-mon[24794]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 741 op/s 2024-03-11T22:29:20.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:20 smithi073 ceph-mon[24794]: Upgrade: Updating mgr.smithi099.rcoixq 2024-03-11T22:29:20.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:20 smithi073 ceph-mon[24794]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:29:20.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 143714 11.83 MB/sec execute 156 sec latency 35.825 ms 2024-03-11T22:29:20.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:20 smithi099 ceph-mon[27157]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 741 op/s 2024-03-11T22:29:20.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:20 smithi099 ceph-mon[27157]: Upgrade: Updating mgr.smithi099.rcoixq 2024-03-11T22:29:20.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:20 smithi099 ceph-mon[27157]: Deploying daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:29:21.484 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:21 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:21.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 144165 11.81 MB/sec execute 157 sec latency 36.537 ms 2024-03-11T22:29:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:21 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:22.494 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:22 smithi073 ceph-mon[24794]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 528 op/s 2024-03-11T22:29:22.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 144576 11.81 MB/sec execute 158 sec latency 41.060 ms 2024-03-11T22:29:22.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:22 smithi099 ceph-mon[27157]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 528 op/s 2024-03-11T22:29:23.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 145036 11.82 MB/sec execute 159 sec latency 37.683 ms 2024-03-11T22:29:24.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 145417 11.84 MB/sec execute 160 sec latency 34.855 ms 2024-03-11T22:29:24.802 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:24 smithi073 ceph-mon[24794]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-03-11T22:29:24.802 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:24 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:24.802 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:24 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:24.803 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:24 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:24 smithi099 ceph-mon[27157]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-03-11T22:29:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:24 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:24 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:24 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:25.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 145800 11.85 MB/sec execute 161 sec latency 43.056 ms 2024-03-11T22:29:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 146168 11.83 MB/sec execute 162 sec latency 42.042 ms 2024-03-11T22:29:26.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:26 smithi073 ceph-mon[24794]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 600 op/s 2024-03-11T22:29:26.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:26 smithi099 ceph-mon[27157]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 600 op/s 2024-03-11T22:29:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 146552 11.79 MB/sec execute 163 sec latency 43.390 ms 2024-03-11T22:29:28.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 147038 11.80 MB/sec execute 164 sec latency 32.732 ms 2024-03-11T22:29:28.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:28 smithi099 ceph-mon[27157]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 618 op/s 2024-03-11T22:29:28.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:28 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:28.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:28 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:28.797 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:28 smithi073 ceph-mon[24794]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 618 op/s 2024-03-11T22:29:28.797 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:28 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:28.797 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:28 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:29.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 147605 11.81 MB/sec execute 165 sec latency 86.078 ms 2024-03-11T22:29:30.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:30 smithi073 ceph-mon[24794]: pgmap v22: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 840 op/s 2024-03-11T22:29:30.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:30 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:30.581 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:30 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:30.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 147928 11.78 MB/sec execute 166 sec latency 56.072 ms 2024-03-11T22:29:30.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:30 smithi099 ceph-mon[27157]: pgmap v22: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 840 op/s 2024-03-11T22:29:30.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:30 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:30.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:30 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:31.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 148188 11.75 MB/sec execute 167 sec latency 59.944 ms 2024-03-11T22:29:32.626 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: pgmap v23: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.1 MiB/s wr, 515 op/s 2024-03-11T22:29:32.626 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq restarted 2024-03-11T22:29:32.626 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:32.626 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:32.626 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:32.627 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:32.627 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:32 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:32.663 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: pgmap v23: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.1 MiB/s wr, 515 op/s 2024-03-11T22:29:32.663 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: Standby manager daemon smithi099.rcoixq restarted 2024-03-11T22:29:32.663 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:32.663 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:32.663 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:32.664 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:32.664 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:32 smithi073 ceph-mon[24794]: from='mgr.? 172.21.15.99:0/2527125053' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:32.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 148549 11.75 MB/sec execute 168 sec latency 48.112 ms 2024-03-11T22:29:33.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 149014 11.78 MB/sec execute 169 sec latency 43.385 ms 2024-03-11T22:29:33.683 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:33 smithi099 ceph-mon[27157]: mgrmap e33: smithi073.pglkvm(active, since 42s), standbys: smithi099.rcoixq 2024-03-11T22:29:33.683 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:33 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:33.683 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:33 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:33.690 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:33 smithi073 ceph-mon[24794]: mgrmap e33: smithi073.pglkvm(active, since 42s), standbys: smithi099.rcoixq 2024-03-11T22:29:33.690 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:33 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:33.690 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:33 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.675 INFO:teuthology.orchestra.run.smithi073.stdout: 5 149467 11.82 MB/sec execute 170 sec latency 29.534 ms 2024-03-11T22:29:34.699 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: pgmap v24: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 768 op/s 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi073.pglkvm"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi073.pglkvm"}]': finished 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi099.rcoixq"}]': finished 2024-03-11T22:29:34.700 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:34 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T22:29:34.944 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:29:34.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: pgmap v24: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 768 op/s 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi073.pglkvm"}]: dispatch 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi073.pglkvm"}]': finished 2024-03-11T22:29:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:34.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi099.rcoixq"}]': finished 2024-03-11T22:29:34.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:34 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T22:29:35.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 149865 11.77 MB/sec execute 171 sec latency 56.992 ms 2024-03-11T22:29:35.704 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:35 smithi073 ceph-mon[24794]: Upgrade: Setting container_image for all mgr 2024-03-11T22:29:35.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:35 smithi099 ceph-mon[27157]: Upgrade: Setting container_image for all mgr 2024-03-11T22:29:36.100 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (2m) 2s ago 11m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (11m) 2s ago 11m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (10m) 8s ago 10m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (108s) 2s ago 11m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (5m) 2s ago 5m 37.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (5m) 8s ago 5m 13.1M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:29:36.101 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (64s) 2s ago 12m 493M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (13s) 8s ago 10m 66.5M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (12m) 2s ago 12m 57.7M 2048M 17.2.0 e1d6a67b021e 1085a4e90bc1 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (10m) 8s ago 10m 42.2M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (5m) 2s ago 5m 272M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (4m) 8s ago 4m 83.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:29:36.102 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (105s) 2s ago 11m 9.91M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (76s) 8s ago 10m 17.1M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (9m) 2s ago 9m 504M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (9m) 2s ago 9m 435M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (8m) 2s ago 8m 329M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (8m) 2s ago 8m 469M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:29:36.103 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (8m) 8s ago 8m 481M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:29:36.104 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (7m) 8s ago 7m 556M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:29:36.104 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (7m) 8s ago 7m 473M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:29:36.104 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (6m) 8s ago 6m 370M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:29:36.104 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (19s) 2s ago 10m 49.0M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:29:36.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 150209 11.73 MB/sec execute 172 sec latency 49.343 ms 2024-03-11T22:29:36.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:36 smithi073 ceph-mon[24794]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:36.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:36 smithi073 ceph-mon[24794]: pgmap v25: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-03-11T22:29:36.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:36 smithi073 ceph-mon[24794]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:36.713 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:36 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:29:36.750 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:29:36.751 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:29:36.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:36 smithi099 ceph-mon[27157]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:36.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:36 smithi099 ceph-mon[27157]: pgmap v25: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 493 op/s 2024-03-11T22:29:36.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:36 smithi099 ceph-mon[27157]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:36.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:36 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:37.297 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:29:37.297 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:29:37.297 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr" 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "2/23 daemons upgraded", 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading mon daemons", 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:29:37.298 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:29:37.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 150551 11.73 MB/sec execute 173 sec latency 44.124 ms 2024-03-11T22:29:37.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:37.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:37.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:29:37.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:29:37.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:37.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:37 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/1537263606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:37.963 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:29:37.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:37.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:37.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:29:37.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:29:37.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:37.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:37 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/1537263606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:29:38.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 151045 11.73 MB/sec execute 174 sec latency 51.796 ms 2024-03-11T22:29:38.729 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:38 smithi073 ceph-mon[24794]: Upgrade: Updating mon.smithi073 2024-03-11T22:29:38.729 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:38 smithi073 ceph-mon[24794]: Deploying daemon mon.smithi073 on smithi073 2024-03-11T22:29:38.729 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:38 smithi073 ceph-mon[24794]: pgmap v26: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-03-11T22:29:38.729 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:38 smithi073 ceph-mon[24794]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:38.729 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:38 smithi073 ceph-mon[24794]: from='client.? 172.21.15.73:0/26235149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:29:38.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:38 smithi099 ceph-mon[27157]: Upgrade: Updating mon.smithi073 2024-03-11T22:29:38.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:38 smithi099 ceph-mon[27157]: Deploying daemon mon.smithi073 on smithi073 2024-03-11T22:29:38.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:38 smithi099 ceph-mon[27157]: pgmap v26: 97 pgs: 97 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-03-11T22:29:38.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:38 smithi099 ceph-mon[27157]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:29:38.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:38 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/26235149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:29:39.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 151480 11.72 MB/sec execute 175 sec latency 66.277 ms 2024-03-11T22:29:39.737 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:39 smithi073 systemd[1]: Stopping Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:29:40.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:40 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[24790]: 2024-03-11T22:29:40.290+0000 7fbd82337700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi073 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:29:40.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:40 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[24790]: 2024-03-11T22:29:40.290+0000 7fbd82337700 -1 mon.smithi073@0(leader) e2 *** Got Signal Terminated *** 2024-03-11T22:29:40.518 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:40 smithi073 podman[66312]: 2024-03-11 22:29:40.380720704 +0000 UTC m=+0.449846455 container died 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-03-11T22:29:40.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 151823 11.71 MB/sec execute 176 sec latency 33.274 ms 2024-03-11T22:29:41.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 podman[66312]: 2024-03-11 22:29:41.235477028 +0000 UTC m=+1.304602772 container cleanup 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, version=8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux ) 2024-03-11T22:29:41.283 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 bash[66312]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073 2024-03-11T22:29:41.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 152164 11.71 MB/sec execute 177 sec latency 42.779 ms 2024-03-11T22:29:41.845 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 podman[66333]: 2024-03-11 22:29:41.698125005 +0000 UTC m=+1.313969178 container remove 1085a4e90bc1c6a40bce3e42427c41f4063bf012507aa624ac3918faca8920a0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, vcs-type=git, architecture=x86_64, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container) 2024-03-11T22:29:42.096 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service: Deactivated successfully. 2024-03-11T22:29:42.096 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 systemd[1]: Stopped Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:29:42.096 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:41 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service: Consumed 16.533s CPU time. 2024-03-11T22:29:42.359 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 systemd[1]: Starting Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:29:42.610 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 podman[66462]: 2024-03-11 22:29:42.393972624 +0000 UTC m=+0.035239748 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:29:42.610 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 podman[66462]: 2024-03-11 22:29:42.506596622 +0000 UTC m=+0.147863731 container create d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T22:29:42.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 152570 11.72 MB/sec execute 178 sec latency 32.321 ms 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 podman[66462]: 2024-03-11 22:29:42.808806182 +0000 UTC m=+0.450073286 container init d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 podman[66462]: 2024-03-11 22:29:42.8139006 +0000 UTC m=+0.455167705 container start d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: pidfile_write: ignore empty --pid-file 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: load: jerasure load: lrc 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:29:42.865 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Git sha 0 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: DB SUMMARY 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: DB Session ID: RVSW1JMUVNUSZRIBPXV5 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi073/store.db dir, Total Num: 1, files: 000030.sst 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi073/store.db: 000028.log size: 2822932 ; 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:29:42.866 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:29:42.867 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:29:42.867 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.env: 0x55c075661ae0 2024-03-11T22:29:42.867 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:29:42.867 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.info_log: 0x55c075eeb760 2024-03-11T22:29:42.867 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.statistics: (nil) 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.use_fsync: 0 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:29:42.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.db_log_dir: 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.wal_dir: 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:29:42.869 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.write_buffer_manager: 0x55c075ebf400 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.unordered_write: 0 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:29:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.row_cache: None 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.wal_filter: None 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.wal_compression: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:29:42.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:29:42.872 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_open_files: -1 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Compression algorithms supported: 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kZSTD supported: 0 2024-03-11T22:29:42.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000015 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.merge_operator: 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_filter: None 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:29:42.874 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c075eeb880) 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_type: 0 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_index_type: 0 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_shortening: 1 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: checksum: 4 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: no_block_cache: 0 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache: 0x55c075eef090 2024-03-11T22:29:42.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_options: 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: capacity : 536870912 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: num_shard_bits : 4 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: strict_capacity_limit : 0 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_cache_compressed: (nil) 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: persistent_cache: (nil) 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size: 4096 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_size_deviation: 10 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_restart_interval: 16 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: index_block_restart_interval: 1 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: metadata_block_size: 4096 2024-03-11T22:29:42.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: partition_filters: 0 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: use_delta_encoding: 1 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: filter_policy: bloomfilter 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: whole_key_filtering: 1 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: verify_compression: 0 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: format_version: 5 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_index_compression: 1 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: block_align: 0 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: max_auto_readahead_size: 262144 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: prepopulate_block_cache: 0 2024-03-11T22:29:42.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression: NoCompression 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.num_levels: 7 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:29:42.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:29:42.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:29:42.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:29:42.881 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:29:42.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.ttl: 2592000 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:29:42.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:29:42.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi073/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11811, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5ebe29d6-3596-4d09-8fe2-0918ea937098 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196182849516, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-11T22:29:42.885 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196182860442, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2548692, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11812, "largest_seqno": 13111, "table_properties": {"data_size": 2542353, "index_size": 3912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1413, "raw_key_size": 12334, "raw_average_key_size": 22, "raw_value_size": 2530452, "raw_average_value_size": 4660, "num_data_blocks": 179, "num_entries": 543, "num_filter_entries": 543, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710196182, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5ebe29d6-3596-4d09-8fe2-0918ea937098", "db_session_id": "RVSW1JMUVNUSZRIBPXV5", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196182860544, "job": 1, "event": "recovery_finished"} 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi073/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c075fde000 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: DB pointer 0x55c075fca000 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** DB Stats ** 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:29:42.886 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L0 1/0 2.43 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 451.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: L6 1/0 6.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Sum 2/0 8.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 451.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-03-11T22:29:42.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 451.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 451.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:29:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Cumulative compaction: 0.00 GB write, 168.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Interval compaction: 0.00 GB write, 168.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Block cache BinnedLRUCache@0x55c075eef090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:29:42.889 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: starting mon.smithi073 rank 0 at public addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] at bind addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi073 fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:29:43.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???) e2 preinit fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:29:43.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).mds e7 new map 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).mds e7 print_map 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: e7 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: legacy client fscid: 1 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Filesystem 'foofs' (1) 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: fs_name foofs 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: epoch 5 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: created 2024-03-11T22:24:06.911675+0000 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: modified 2024-03-11T22:24:13.032504+0000 2024-03-11T22:29:43.116 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: tableserver 0 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: root 0 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: session_timeout 60 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: session_autoclose 300 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: max_file_size 1099511627776 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: max_xattr_size 65536 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: required_client_features {} 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: last_failure 0 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: last_failure_osd_epoch 0 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: max_mds 1 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: in 0 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: up {0=14482} 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: failed 2024-03-11T22:29:43.117 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: damaged 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: stopped 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: data_pools [3] 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: metadata_pool 2 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: inline_data disabled 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: balancer 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: bal_rank_mask -1 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: standby_count_wanted 1 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: [mds.foofs.smithi073.lmopce{0:14482} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: Standby daemons: 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: 2024-03-11T22:29:43.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout: [mds.foofs.smithi099.kagazk{-1:24299} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 ceph-mon[66481]: mon.smithi073@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 bash[66462]: d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 2024-03-11T22:29:43.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:42 smithi073 systemd[1]: Started Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:29:43.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 152901 11.73 MB/sec execute 179 sec latency 40.241 ms 2024-03-11T22:29:43.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:29:43.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 502 op/s 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: mon.smithi073 calling monitor election 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: monmap e2: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: mgrmap e33: smithi073.pglkvm(active, since 53s), standbys: smithi099.rcoixq 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: overall HEALTH_OK 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: from='mgr.14792 ' entity='' 2024-03-11T22:29:43.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:43 smithi073 ceph-mon[66481]: mgrmap e34: smithi073.pglkvm(active, since 53s), standbys: smithi099.rcoixq 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: from='mgr.14792 172.21.15.73:0/2801057607' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.1 MiB/s wr, 502 op/s 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: mon.smithi073 calling monitor election 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: monmap e2: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:29:43.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:29:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: mgrmap e33: smithi073.pglkvm(active, since 53s), standbys: smithi099.rcoixq 2024-03-11T22:29:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: overall HEALTH_OK 2024-03-11T22:29:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: from='mgr.14792 ' entity='' 2024-03-11T22:29:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:43 smithi099 ceph-mon[27157]: mgrmap e34: smithi073.pglkvm(active, since 53s), standbys: smithi099.rcoixq 2024-03-11T22:29:44.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 153138 11.70 MB/sec execute 180 sec latency 54.089 ms 2024-03-11T22:29:45.657 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:45 smithi073 ceph-mon[66481]: mgrmap e35: smithi073.pglkvm(active, since 54s), standbys: smithi099.rcoixq 2024-03-11T22:29:45.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 153617 11.67 MB/sec execute 181 sec latency 38.596 ms 2024-03-11T22:29:45.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:45 smithi099 ceph-mon[27157]: mgrmap e35: smithi073.pglkvm(active, since 54s), standbys: smithi099.rcoixq 2024-03-11T22:29:46.678 INFO:teuthology.orchestra.run.smithi073.stdout: 5 154086 11.68 MB/sec execute 182 sec latency 32.583 ms 2024-03-11T22:29:47.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 154536 11.67 MB/sec execute 183 sec latency 38.466 ms 2024-03-11T22:29:48.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 155026 11.67 MB/sec execute 184 sec latency 49.310 ms 2024-03-11T22:29:49.676 INFO:teuthology.orchestra.run.smithi073.stdout: 5 155404 11.66 MB/sec execute 185 sec latency 30.743 ms 2024-03-11T22:29:50.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 155816 11.67 MB/sec execute 186 sec latency 36.743 ms 2024-03-11T22:29:51.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 156201 11.69 MB/sec execute 187 sec latency 42.441 ms 2024-03-11T22:29:51.985 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: Standby manager daemon smithi099.rcoixq restarted 2024-03-11T22:29:51.985 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:51.985 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:51.986 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:51.986 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:51.986 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:51 smithi073 ceph-mon[66481]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:51.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq restarted 2024-03-11T22:29:51.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: Standby manager daemon smithi099.rcoixq started 2024-03-11T22:29:51.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/crt"}]: dispatch 2024-03-11T22:29:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:29:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi099.rcoixq/key"}]: dispatch 2024-03-11T22:29:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:51 smithi099 ceph-mon[27157]: from='mgr.? 172.21.15.99:0/511350993' entity='mgr.smithi099.rcoixq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:29:52.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 156526 11.69 MB/sec execute 188 sec latency 36.118 ms 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:52 smithi073 ceph-mon[66481]: mgrmap e36: smithi073.pglkvm(active, since 61s), standbys: smithi099.rcoixq 2024-03-11T22:29:53.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:52 smithi099 ceph-mon[27157]: mgrmap e36: smithi073.pglkvm(active, since 61s), standbys: smithi099.rcoixq 2024-03-11T22:29:53.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 156921 11.66 MB/sec execute 189 sec latency 64.154 ms 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: mgrmap e37: smithi073.pglkvm(active, starting, since 0.00769582s), standbys: smithi099.rcoixq 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:29:53.856 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:53.857 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:53.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:53.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:29:53.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:29:53.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:29:53.858 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:53 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099[27139]: 2024-03-11T22:29:53.563+0000 7f0a4174f700 -1 mon.smithi099@1(peon).osd e74 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: Active manager daemon smithi073.pglkvm restarted 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: Activating manager daemon smithi073.pglkvm 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: mgrmap e37: smithi073.pglkvm(active, starting, since 0.00769582s), standbys: smithi099.rcoixq 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:29:53.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi073.pglkvm", "id": "smithi073.pglkvm"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr metadata", "who": "smithi099.rcoixq", "id": "smithi099.rcoixq"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:29:53.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: Manager daemon smithi073.pglkvm is now available 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:29:53.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:53 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi073.pglkvm/trash_purge_schedule"}]: dispatch 2024-03-11T22:29:54.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 157205 11.62 MB/sec execute 190 sec latency 69.078 ms 2024-03-11T22:29:54.877 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: failed to encode map e74 with expected crc 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: failed to encode map e74 with expected crc 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: failed to encode map e74 with expected crc 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: [11/Mar/2024:22:29:54] ENGINE Bus STARTING 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: [11/Mar/2024:22:29:54] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: [11/Mar/2024:22:29:54] ENGINE Serving on http://172.21.15.73:8765 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: [11/Mar/2024:22:29:54] ENGINE Bus STARTED 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: mgrmap e38: smithi073.pglkvm(active, since 1.02359s), standbys: smithi099.rcoixq 2024-03-11T22:29:54.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:54 smithi073 ceph-mon[66481]: pgmap v3: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: failed to encode map e74 with expected crc 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: failed to encode map e74 with expected crc 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: failed to encode map e74 with expected crc 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: [11/Mar/2024:22:29:54] ENGINE Bus STARTING 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: [11/Mar/2024:22:29:54] ENGINE Serving on https://172.21.15.73:7150 2024-03-11T22:29:55.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: [11/Mar/2024:22:29:54] ENGINE Serving on http://172.21.15.73:8765 2024-03-11T22:29:55.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: [11/Mar/2024:22:29:54] ENGINE Bus STARTED 2024-03-11T22:29:55.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: mgrmap e38: smithi073.pglkvm(active, since 1.02359s), standbys: smithi099.rcoixq 2024-03-11T22:29:55.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:54 smithi099 ceph-mon[27157]: pgmap v3: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:55.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 157522 11.61 MB/sec execute 191 sec latency 46.254 ms 2024-03-11T22:29:56.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 157824 11.59 MB/sec execute 192 sec latency 74.488 ms 2024-03-11T22:29:56.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:56 smithi073 ceph-mon[66481]: pgmap v4: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:56.881 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:56 smithi099 ceph-mon[27157]: pgmap v4: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:57.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 158204 11.58 MB/sec execute 193 sec latency 42.984 ms 2024-03-11T22:29:57.760 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:57 smithi073 ceph-mon[66481]: mgrmap e39: smithi073.pglkvm(active, since 3s), standbys: smithi099.rcoixq 2024-03-11T22:29:57.913 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:57 smithi099 ceph-mon[27157]: mgrmap e39: smithi073.pglkvm(active, since 3s), standbys: smithi099.rcoixq 2024-03-11T22:29:58.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 158529 11.56 MB/sec execute 194 sec latency 49.395 ms 2024-03-11T22:29:58.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:58 smithi073 ceph-mon[66481]: pgmap v5: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:58.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:58 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:58.773 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:58 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:58.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:58 smithi099 ceph-mon[27157]: pgmap v5: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:29:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:58 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:58 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:59.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 158762 11.51 MB/sec execute 195 sec latency 50.692 ms 2024-03-11T22:29:59.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:59 smithi073 ceph-mon[66481]: mgrmap e40: smithi073.pglkvm(active, since 5s), standbys: smithi099.rcoixq 2024-03-11T22:29:59.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:59 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:59.798 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:29:59 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:59.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:59 smithi099 ceph-mon[27157]: mgrmap e40: smithi073.pglkvm(active, since 5s), standbys: smithi099.rcoixq 2024-03-11T22:29:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:59 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:29:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:29:59 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:00.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 159235 11.54 MB/sec execute 196 sec latency 35.567 ms 2024-03-11T22:30:00.818 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:00 smithi073 ceph-mon[66481]: pgmap v6: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:30:00.818 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:00 smithi073 ceph-mon[66481]: overall HEALTH_OK 2024-03-11T22:30:00.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:00 smithi099 ceph-mon[27157]: pgmap v6: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-03-11T22:30:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:00 smithi099 ceph-mon[27157]: overall HEALTH_OK 2024-03-11T22:30:01.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 159671 11.56 MB/sec execute 197 sec latency 52.656 ms 2024-03-11T22:30:02.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 160098 11.60 MB/sec execute 198 sec latency 38.124 ms 2024-03-11T22:30:02.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:02 smithi099 ceph-mon[27157]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-03-11T22:30:02.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:02 smithi073 ceph-mon[66481]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-03-11T22:30:03.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 160504 11.56 MB/sec execute 199 sec latency 34.165 ms 2024-03-11T22:30:04.400 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:04 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.400 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:04 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.400 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:04 smithi073 ceph-mon[66481]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 276 op/s 2024-03-11T22:30:04.401 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:04 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.401 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:04 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:04 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:04 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:04 smithi099 ceph-mon[27157]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 276 op/s 2024-03-11T22:30:04.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:04 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:04 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:04.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 160866 11.53 MB/sec execute 200 sec latency 34.627 ms 2024-03-11T22:30:05.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 161213 11.53 MB/sec execute 201 sec latency 45.998 ms 2024-03-11T22:30:05.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:05.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:30:05.915 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:05 smithi073 ceph-mon[66481]: pgmap v9: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.conf 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:30:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:30:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:30:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:05 smithi099 ceph-mon[27157]: pgmap v9: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-03-11T22:30:06.677 INFO:teuthology.orchestra.run.smithi073.stdout: 5 161701 11.53 MB/sec execute 202 sec latency 33.047 ms 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:07.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:07.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:07 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: Updating smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: Updating smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:07.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:07.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:07.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:07 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-11T22:30:07.680 INFO:teuthology.orchestra.run.smithi073.stdout: 5 162143 11.52 MB/sec execute 203 sec latency 59.083 ms 2024-03-11T22:30:08.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:08 smithi073 ceph-mon[66481]: pgmap v10: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-03-11T22:30:08.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:08 smithi099 ceph-mon[27157]: pgmap v10: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-03-11T22:30:08.508 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:30:08.680 INFO:teuthology.orchestra.run.smithi073.stdout: 5 162483 11.50 MB/sec execute 204 sec latency 41.039 ms 2024-03-11T22:30:09.238 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: Upgrade: Updating mon.smithi099 2024-03-11T22:30:09.238 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:09.238 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:09.239 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:09.239 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:09.239 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: Deploying daemon mon.smithi099 on smithi099 2024-03-11T22:30:09.239 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:09.239 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (2m) 6s ago 11m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (11m) 6s ago 11m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (11m) 11s ago 11m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (2m) 6s ago 11m 45.6M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:30:09.487 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (5m) 6s ago 5m 37.7M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (5m) 11s ago 5m 13.3M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (98s) 6s ago 13m 478M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (46s) 11s ago 11m 447M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (27s) 6s ago 13m 30.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (11m) 11s ago 11m 49.1M 2048M 17.2.0 e1d6a67b021e 803548aea880 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (5m) 6s ago 5m 279M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (5m) 11s ago 5m 83.3M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (2m) 6s ago 11m 9911k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (109s) 11s ago 10m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (10m) 6s ago 10m 558M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (9m) 6s ago 9m 479M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (9m) 6s ago 9m 357M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (8m) 6s ago 8m 498M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:30:09.488 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (8m) 11s ago 8m 508M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:30:09.489 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (8m) 11s ago 8m 324M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:30:09.489 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (7m) 11s ago 7m 513M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:30:09.489 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (7m) 11s ago 7m 404M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:30:09.489 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (53s) 6s ago 10m 55.6M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:30:09.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: Upgrade: Updating mon.smithi099 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='mgr.34104 ' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: Deploying daemon mon.smithi099 on smithi099 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:09.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:09 smithi099 ceph-mon[27157]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:09.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 162832 11.51 MB/sec execute 205 sec latency 41.636 ms 2024-03-11T22:30:10.070 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:10.071 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T22:30:10.072 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:30:10.073 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:30:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:10 smithi099 ceph-mon[27157]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:10 smithi099 ceph-mon[27157]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:10 smithi099 ceph-mon[27157]: pgmap v11: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-03-11T22:30:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:10 smithi099 ceph-mon[27157]: from='client.? 172.21.15.73:0/291840736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:10.250 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:10 smithi073 ceph-mon[66481]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:10.250 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:10 smithi073 ceph-mon[66481]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:10.250 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:10 smithi073 ceph-mon[66481]: pgmap v11: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-03-11T22:30:10.250 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:10 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/291840736' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr" 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "3/23 daemons upgraded", 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading mon daemons", 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:30:10.576 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:30:10.680 INFO:teuthology.orchestra.run.smithi073.stdout: 5 163221 11.53 MB/sec execute 206 sec latency 54.127 ms 2024-03-11T22:30:10.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:10 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:30:11.167 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:30:11.324 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099[27139]: 2024-03-11T22:30:11.001+0000 7f0a46f5a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:30:11.324 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099[27139]: 2024-03-11T22:30:11.001+0000 7f0a46f5a700 -1 mon.smithi099@1(peon) e2 *** Got Signal Terminated *** 2024-03-11T22:30:11.324 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 podman[53046]: 2024-03-11 22:30:11.031957537 +0000 UTC m=+0.618428839 container died 803548aea8809419bb1964a1ec5b4ac979234f840a6df80644303bed96d539f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, version=8, release=754, GIT_CLEAN=True, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870) 2024-03-11T22:30:11.680 INFO:teuthology.orchestra.run.smithi073.stdout: 5 163480 11.52 MB/sec execute 207 sec latency 68.312 ms 2024-03-11T22:30:11.700 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 podman[53046]: 2024-03-11 22:30:11.418529201 +0000 UTC m=+1.005000512 container cleanup 803548aea8809419bb1964a1ec5b4ac979234f840a6df80644303bed96d539f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, ceph=True, name=centos-stream, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container) 2024-03-11T22:30:11.700 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 bash[53046]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099 2024-03-11T22:30:11.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 podman[53060]: 2024-03-11 22:30:11.695437876 +0000 UTC m=+0.666770651 container remove 803548aea8809419bb1964a1ec5b4ac979234f840a6df80644303bed96d539f0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, vcs-type=git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, release=754, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-03-11T22:30:11.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi099.service: Deactivated successfully. 2024-03-11T22:30:11.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 systemd[1]: Stopped Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:30:11.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:11 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi099.service: Consumed 7.106s CPU time. 2024-03-11T22:30:12.376 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 systemd[1]: Starting Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:30:12.376 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 podman[53185]: 2024-03-11 22:30:12.232845052 +0000 UTC m=+0.022319584 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:30:12.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 podman[53185]: 2024-03-11 22:30:12.372001798 +0000 UTC m=+0.161476321 container create bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T22:30:12.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 podman[53185]: 2024-03-11 22:30:12.611457838 +0000 UTC m=+0.400932365 container init bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:30:12.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 podman[53185]: 2024-03-11 22:30:12.616143901 +0000 UTC m=+0.405618433 container start bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T22:30:12.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 163708 11.50 MB/sec execute 208 sec latency 64.178 ms 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: pidfile_write: ignore empty --pid-file 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: load: jerasure load: lrc 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:30:12.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Git sha 0 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: DB SUMMARY 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: DB Session ID: 8Z6O6V8QAY9VYP0SI4O8 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 1, files: 000024.sst 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 000022.log size: 7436996 ; 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:30:12.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.env: 0x564c12d0dae0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.info_log: 0x564c139d7760 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.statistics: (nil) 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.use_fsync: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:30:12.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.db_log_dir: 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.wal_dir: 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.write_buffer_manager: 0x564c139ab400 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:30:12.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.unordered_write: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.row_cache: None 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.wal_filter: None 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:30:12.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.wal_compression: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:30:12.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_open_files: -1 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:30:13.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Compression algorithms supported: 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kZSTD supported: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:30:13.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000009 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.merge_operator: 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_filter: None 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564c139d7880) 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_type: 0 2024-03-11T22:30:13.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: data_block_index_type: 0 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_shortening: 1 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: checksum: 4 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: no_block_cache: 0 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache: 0x564c139db090 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_options: 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: capacity : 536870912 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: num_shard_bits : 4 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: strict_capacity_limit : 0 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_cache_compressed: (nil) 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: persistent_cache: (nil) 2024-03-11T22:30:13.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_size: 4096 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_size_deviation: 10 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_restart_interval: 16 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: index_block_restart_interval: 1 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: metadata_block_size: 4096 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: partition_filters: 0 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: use_delta_encoding: 1 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: filter_policy: bloomfilter 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: whole_key_filtering: 1 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: verify_compression: 0 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: format_version: 5 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: enable_index_compression: 1 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: block_align: 0 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: max_auto_readahead_size: 262144 2024-03-11T22:30:13.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: prepopulate_block_cache: 0 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression: NoCompression 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.num_levels: 7 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:30:13.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:30:13.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:30:13.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:30:13.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:30:13.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:30:13.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:30:13.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:30:13.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:30:13.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:30:13.010 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.ttl: 2592000 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:30:13.011 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12185, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3e188185-4663-4bac-902d-c0db1e698852 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196212648100, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196212666772, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4477294, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12190, "largest_seqno": 14181, "table_properties": {"data_size": 4468761, "index_size": 5593, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17452, "raw_average_key_size": 22, "raw_value_size": 4452077, "raw_average_value_size": 5812, "num_data_blocks": 255, "num_entries": 766, "num_filter_entries": 766, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710196212, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3e188185-4663-4bac-902d-c0db1e698852", "db_session_id": "8Z6O6V8QAY9VYP0SI4O8", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710196212666921, "job": 1, "event": "recovery_finished"} 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi099/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564c13aca000 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: DB pointer 0x564c13ab6000 2024-03-11T22:30:13.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** DB Stats ** 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: L0 1/0 4.27 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-11T22:30:13.013 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: L6 1/0 6.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Sum 2/0 10.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 486.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** Compaction Stats [default] ** 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 486.3 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:30:13.014 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Cumulative compaction: 0.00 GB write, 194.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Interval compaction: 0.00 GB write, 194.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Block cache BinnedLRUCache@0x564c139db090#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: starting mon.smithi099 rank 1 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi099 fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???) e2 preinit fsid f2f121a8-dff4-11ee-95c8-87774f69a715 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).mds e7 new map 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).mds e7 print_map 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: e7 2024-03-11T22:30:13.015 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: legacy client fscid: 1 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Filesystem 'foofs' (1) 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: fs_name foofs 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: epoch 5 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: created 2024-03-11T22:24:06.911675+0000 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: modified 2024-03-11T22:24:13.032504+0000 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: tableserver 0 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: root 0 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: session_timeout 60 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: session_autoclose 300 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: max_file_size 1099511627776 2024-03-11T22:30:13.016 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: max_xattr_size 65536 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: required_client_features {} 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: last_failure 0 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: last_failure_osd_epoch 0 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: max_mds 1 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: in 0 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: up {0=14482} 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: failed 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: damaged 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: stopped 2024-03-11T22:30:13.017 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: data_pools [3] 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: metadata_pool 2 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: inline_data disabled 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: balancer 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: bal_rank_mask -1 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: standby_count_wanted 1 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: [mds.foofs.smithi073.lmopce{0:14482} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.73:6834/3872767365,v1:172.21.15.73:6835/3872767365] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: Standby daemons: 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout: [mds.foofs.smithi099.kagazk{-1:24299} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).osd e74 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:30:13.018 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:30:13.019 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-03-11T22:30:13.019 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 ceph-mon[53203]: mon.smithi099@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-03-11T22:30:13.019 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 bash[53185]: bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb 2024-03-11T22:30:13.019 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:12 smithi099 systemd[1]: Started Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:30:13.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 164001 11.46 MB/sec execute 209 sec latency 53.692 ms 2024-03-11T22:30:14.356 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:14 smithi073 ceph-mon[66481]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:14.356 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:14 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:30:14.356 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:14 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:30:14.356 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:14 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2437253972' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:30:14.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 164216 11.42 MB/sec execute 210 sec latency 75.945 ms 2024-03-11T22:30:14.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:14 smithi099 ceph-mon[53203]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:14.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:14 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:30:14.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:14 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:30:14.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:14 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2437253972' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:30:15.366 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: mon.smithi073 calling monitor election 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: mon.smithi099 calling monitor election 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: monmap e3: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: mgrmap e40: smithi073.pglkvm(active, since 20s), standbys: smithi099.rcoixq 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: overall HEALTH_OK 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:15.367 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:15.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 164590 11.41 MB/sec execute 211 sec latency 41.823 ms 2024-03-11T22:30:15.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi073"}]: dispatch 2024-03-11T22:30:15.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: mon.smithi073 calling monitor election 2024-03-11T22:30:15.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: mon.smithi099 calling monitor election 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: mon.smithi073 is new leader, mons smithi073,smithi099 in quorum (ranks 0,1) 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: monmap e3: 2 mons at {smithi073=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: mgrmap e40: smithi073.pglkvm(active, since 20s), standbys: smithi099.rcoixq 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: overall HEALTH_OK 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:15.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:16.402 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:16 smithi073 ceph-mon[66481]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.0 MiB/s wr, 666 op/s 2024-03-11T22:30:16.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 165047 11.42 MB/sec execute 212 sec latency 37.557 ms 2024-03-11T22:30:16.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:16 smithi099 ceph-mon[53203]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.0 MiB/s wr, 666 op/s 2024-03-11T22:30:17.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 165539 11.42 MB/sec execute 213 sec latency 31.344 ms 2024-03-11T22:30:18.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 166018 11.41 MB/sec execute 214 sec latency 28.713 ms 2024-03-11T22:30:18.691 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:18 smithi073 ceph-mon[66481]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-11T22:30:18.691 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:18 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:18.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:18 smithi099 ceph-mon[53203]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-11T22:30:18.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:18 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:19.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 166460 11.43 MB/sec execute 215 sec latency 33.797 ms 2024-03-11T22:30:19.698 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:19 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:19.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:19 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:20.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 166941 11.47 MB/sec execute 216 sec latency 30.729 ms 2024-03-11T22:30:20.710 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:20 smithi073 ceph-mon[66481]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-11T22:30:20.710 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:20 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:20.710 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:20 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:20.959 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:20 smithi099 ceph-mon[53203]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 412 op/s 2024-03-11T22:30:20.959 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:20 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:20.959 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:20 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:21.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 167297 11.47 MB/sec execute 217 sec latency 43.683 ms 2024-03-11T22:30:22.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 167671 11.43 MB/sec execute 218 sec latency 44.051 ms 2024-03-11T22:30:22.694 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:22 smithi099 ceph-mon[53203]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 687 op/s 2024-03-11T22:30:22.740 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:22 smithi073 ceph-mon[66481]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 687 op/s 2024-03-11T22:30:23.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 167875 11.40 MB/sec execute 219 sec latency 41.831 ms 2024-03-11T22:30:24.032 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:23 smithi073 ceph-mon[66481]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 446 op/s 2024-03-11T22:30:24.032 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:24.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:23 smithi099 ceph-mon[53203]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 446 op/s 2024-03-11T22:30:24.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:24.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 168148 11.38 MB/sec execute 220 sec latency 42.079 ms 2024-03-11T22:30:25.147 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: Reconfiguring mon.smithi073 (monmap changed)... 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:25.148 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:25 smithi073 ceph-mon[66481]: Reconfiguring daemon mon.smithi073 on smithi073 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: Reconfiguring mon.smithi073 (monmap changed)... 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:25.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:25.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:25 smithi099 ceph-mon[53203]: Reconfiguring daemon mon.smithi073 on smithi073 2024-03-11T22:30:25.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 168444 11.37 MB/sec execute 221 sec latency 64.296 ms 2024-03-11T22:30:26.170 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:26 smithi073 ceph-mon[66481]: pgmap v19: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 668 op/s 2024-03-11T22:30:26.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:26 smithi099 ceph-mon[53203]: pgmap v19: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 668 op/s 2024-03-11T22:30:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 168802 11.36 MB/sec execute 222 sec latency 48.129 ms 2024-03-11T22:30:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 169143 11.34 MB/sec execute 223 sec latency 45.839 ms 2024-03-11T22:30:27.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:27.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:27.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:30:27.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:30:27.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi073.pglkvm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:30:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:30:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:28.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 169416 11.31 MB/sec execute 224 sec latency 54.593 ms 2024-03-11T22:30:28.718 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:28 smithi073 ceph-mon[66481]: Reconfiguring mgr.smithi073.pglkvm (monmap changed)... 2024-03-11T22:30:28.718 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:28 smithi073 ceph-mon[66481]: Reconfiguring daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:30:28.718 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:28 smithi073 ceph-mon[66481]: pgmap v20: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-11T22:30:28.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:28 smithi099 ceph-mon[53203]: Reconfiguring mgr.smithi073.pglkvm (monmap changed)... 2024-03-11T22:30:28.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:28 smithi099 ceph-mon[53203]: Reconfiguring daemon mgr.smithi073.pglkvm on smithi073 2024-03-11T22:30:28.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:28 smithi099 ceph-mon[53203]: pgmap v20: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-11T22:30:29.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 169783 11.32 MB/sec execute 225 sec latency 42.525 ms 2024-03-11T22:30:30.233 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: Reconfiguring crash.smithi073 (monmap changed)... 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: Reconfiguring daemon crash.smithi073 on smithi073 2024-03-11T22:30:30.234 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:30 smithi073 ceph-mon[66481]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: Reconfiguring crash.smithi073 (monmap changed)... 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: Reconfiguring daemon crash.smithi073 on smithi073 2024-03-11T22:30:30.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:30 smithi099 ceph-mon[53203]: pgmap v21: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-11T22:30:30.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 170080 11.30 MB/sec execute 226 sec latency 41.616 ms 2024-03-11T22:30:31.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 170410 11.32 MB/sec execute 227 sec latency 55.681 ms 2024-03-11T22:30:32.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 170667 11.31 MB/sec execute 228 sec latency 46.247 ms 2024-03-11T22:30:32.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:32 smithi073 ceph-mon[66481]: pgmap v22: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-03-11T22:30:32.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:32.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:32.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:30:32.802 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:32 smithi099 ceph-mon[53203]: pgmap v22: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-03-11T22:30:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:30:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:33.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 170937 11.29 MB/sec execute 229 sec latency 55.982 ms 2024-03-11T22:30:33.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:33 smithi073 ceph-mon[66481]: Reconfiguring osd.0 (monmap changed)... 2024-03-11T22:30:33.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:33 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.0 on smithi073 2024-03-11T22:30:33.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:33 smithi099 ceph-mon[53203]: Reconfiguring osd.0 (monmap changed)... 2024-03-11T22:30:33.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:33 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.0 on smithi073 2024-03-11T22:30:34.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 171182 11.25 MB/sec execute 230 sec latency 50.358 ms 2024-03-11T22:30:34.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:34 smithi073 ceph-mon[66481]: pgmap v23: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 415 op/s 2024-03-11T22:30:34.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:34.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:34.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:30:34.841 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:34 smithi099 ceph-mon[53203]: pgmap v23: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 415 op/s 2024-03-11T22:30:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:30:34.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:35.681 INFO:teuthology.orchestra.run.smithi073.stdout: 5 171386 11.22 MB/sec execute 231 sec latency 51.023 ms 2024-03-11T22:30:35.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:35 smithi073 ceph-mon[66481]: Reconfiguring osd.1 (monmap changed)... 2024-03-11T22:30:35.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:35 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.1 on smithi073 2024-03-11T22:30:35.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:35 smithi099 ceph-mon[53203]: Reconfiguring osd.1 (monmap changed)... 2024-03-11T22:30:35.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:35 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.1 on smithi073 2024-03-11T22:30:36.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 171664 11.20 MB/sec execute 232 sec latency 50.734 ms 2024-03-11T22:30:36.887 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:36 smithi073 ceph-mon[66481]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.0 MiB/s wr, 586 op/s 2024-03-11T22:30:36.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:36 smithi099 ceph-mon[53203]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.0 MiB/s wr, 586 op/s 2024-03-11T22:30:37.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 171962 11.19 MB/sec execute 233 sec latency 47.725 ms 2024-03-11T22:30:38.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:38.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:38.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: Reconfiguring osd.2 (monmap changed)... 2024-03-11T22:30:38.168 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:30:38.168 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:38.168 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.2 on smithi073 2024-03-11T22:30:38.168 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:38 smithi073 ceph-mon[66481]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: Reconfiguring osd.2 (monmap changed)... 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.2 on smithi073 2024-03-11T22:30:38.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:38 smithi099 ceph-mon[53203]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-03-11T22:30:38.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 172273 11.18 MB/sec execute 234 sec latency 61.596 ms 2024-03-11T22:30:39.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:39.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:39.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 172575 11.15 MB/sec execute 235 sec latency 58.880 ms 2024-03-11T22:30:40.229 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:40 smithi073 ceph-mon[66481]: pgmap v26: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-03-11T22:30:40.229 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:40.230 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:40.230 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:30:40.230 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:40.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:40 smithi099 ceph-mon[53203]: pgmap v26: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 365 op/s 2024-03-11T22:30:40.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:40.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:40.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:30:40.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:40.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 172836 11.13 MB/sec execute 236 sec latency 63.405 ms 2024-03-11T22:30:41.270 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:41 smithi073 ceph-mon[66481]: Reconfiguring osd.3 (monmap changed)... 2024-03-11T22:30:41.271 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:41 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.3 on smithi073 2024-03-11T22:30:41.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:41 smithi099 ceph-mon[53203]: Reconfiguring osd.3 (monmap changed)... 2024-03-11T22:30:41.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:41 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.3 on smithi073 2024-03-11T22:30:41.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 173161 11.12 MB/sec execute 237 sec latency 59.444 ms 2024-03-11T22:30:41.770 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:30:42.284 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:42 smithi073 ceph-mon[66481]: pgmap v27: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 544 op/s 2024-03-11T22:30:42.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:42 smithi099 ceph-mon[53203]: pgmap v27: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.2 MiB/s wr, 544 op/s 2024-03-11T22:30:42.682 INFO:teuthology.orchestra.run.smithi073.stdout: 5 173499 11.12 MB/sec execute 238 sec latency 53.379 ms 2024-03-11T22:30:42.713 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:30:42.713 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (3m) 39s ago 12m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (12m) 39s ago 12m 6395k - 17.2.0 e1d6a67b021e 1bdd811890f0 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (11m) 24s ago 11m 6405k - 17.2.0 e1d6a67b021e ccdc4e5b5922 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (2m) 39s ago 12m 45.6M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (6m) 39s ago 6m 37.7M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (6m) 24s ago 6m 13.6M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (2m) 39s ago 13m 478M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (79s) 24s ago 11m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (60s) 39s ago 13m 30.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (30s) 24s ago 11m 24.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (6m) 39s ago 6m 279M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (6m) 24s ago 6m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (2m) 39s ago 12m 9911k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (2m) 24s ago 11m 19.9M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:30:42.714 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (10m) 39s ago 10m 558M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (10m) 39s ago 10m 479M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (9m) 39s ago 9m 357M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (9m) 39s ago 9m 498M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (9m) 24s ago 9m 528M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (8m) 24s ago 8m 336M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (8m) 24s ago 8m 539M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (7m) 24s ago 7m 446M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:30:42.715 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (86s) 39s ago 11m 55.6M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:30:43.342 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:43.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:43.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:43.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:43.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:30:43.343 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:43.475 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:30:43.475 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:30:43.476 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:30:43.476 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:43.476 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:30:43.476 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:30:43.476 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:30:43.477 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:30:43.478 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:30:43.478 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-11T22:30:43.478 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T22:30:43.478 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:30:43.478 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:30:43.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:43.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 173839 11.14 MB/sec execute 239 sec latency 53.817 ms 2024-03-11T22:30:44.036 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:30:44.036 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:30:44.036 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr" 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "4/23 daemons upgraded", 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading mon daemons", 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:30:44.037 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:30:44.362 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:44 smithi073 ceph-mon[66481]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:44.363 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:44 smithi073 ceph-mon[66481]: Reconfiguring mds.foofs.smithi073.lmopce (monmap changed)... 2024-03-11T22:30:44.363 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:44 smithi073 ceph-mon[66481]: Reconfiguring daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:30:44.363 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:44 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1469544456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:44.363 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:44 smithi073 ceph-mon[66481]: pgmap v28: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 350 op/s 2024-03-11T22:30:44.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:44 smithi099 ceph-mon[53203]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:44.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:44 smithi099 ceph-mon[53203]: Reconfiguring mds.foofs.smithi073.lmopce (monmap changed)... 2024-03-11T22:30:44.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:44 smithi099 ceph-mon[53203]: Reconfiguring daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:30:44.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:44 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1469544456' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:30:44.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:44 smithi099 ceph-mon[53203]: pgmap v28: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 350 op/s 2024-03-11T22:30:44.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 174180 11.14 MB/sec execute 240 sec latency 48.493 ms 2024-03-11T22:30:44.693 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:30:45.375 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:45 smithi073 ceph-mon[66481]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:45.375 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:45 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1473944737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:30:45.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:45 smithi099 ceph-mon[53203]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:30:45.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:45 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1473944737' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:30:45.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 174517 11.12 MB/sec execute 241 sec latency 32.963 ms 2024-03-11T22:30:46.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: Reconfiguring crash.smithi099 (monmap changed)... 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:46.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:46 smithi073 ceph-mon[66481]: Reconfiguring daemon crash.smithi099 on smithi099 2024-03-11T22:30:46.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 174818 11.09 MB/sec execute 242 sec latency 51.424 ms 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: Reconfiguring crash.smithi099 (monmap changed)... 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:46.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:46 smithi099 ceph-mon[53203]: Reconfiguring daemon crash.smithi099 on smithi099 2024-03-11T22:30:47.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 175044 11.07 MB/sec execute 243 sec latency 40.051 ms 2024-03-11T22:30:48.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 175375 11.07 MB/sec execute 244 sec latency 54.937 ms 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:30:48.709 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:48.923 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-11T22:30:48.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:48.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:48.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.rcoixq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:30:48.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:30:48.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:49.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 175738 11.05 MB/sec execute 245 sec latency 55.151 ms 2024-03-11T22:30:49.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:49 smithi073 ceph-mon[66481]: Reconfiguring mgr.smithi099.rcoixq (monmap changed)... 2024-03-11T22:30:49.746 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:49 smithi073 ceph-mon[66481]: Reconfiguring daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:30:49.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:49 smithi099 ceph-mon[53203]: Reconfiguring mgr.smithi099.rcoixq (monmap changed)... 2024-03-11T22:30:49.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:49 smithi099 ceph-mon[53203]: Reconfiguring daemon mgr.smithi099.rcoixq on smithi099 2024-03-11T22:30:50.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 176127 11.04 MB/sec execute 246 sec latency 46.074 ms 2024-03-11T22:30:50.755 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-11T22:30:50.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:50.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:50.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:50.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:50.756 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:50.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-11T22:30:50.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:50.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:50.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:30:50.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:30:50.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:51.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 176381 11.02 MB/sec execute 247 sec latency 75.266 ms 2024-03-11T22:30:51.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:51 smithi073 ceph-mon[66481]: Reconfiguring mon.smithi099 (monmap changed)... 2024-03-11T22:30:51.767 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:51 smithi073 ceph-mon[66481]: Reconfiguring daemon mon.smithi099 on smithi099 2024-03-11T22:30:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:51 smithi099 ceph-mon[53203]: Reconfiguring mon.smithi099 (monmap changed)... 2024-03-11T22:30:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:51 smithi099 ceph-mon[53203]: Reconfiguring daemon mon.smithi099 on smithi099 2024-03-11T22:30:52.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 176790 11.03 MB/sec execute 248 sec latency 37.821 ms 2024-03-11T22:30:52.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:52 smithi099 ceph-mon[53203]: pgmap v32: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.5 MiB/s wr, 573 op/s 2024-03-11T22:30:52.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:52.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:52.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:30:52.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:52.777 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:52 smithi073 ceph-mon[66481]: pgmap v32: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.5 MiB/s wr, 573 op/s 2024-03-11T22:30:52.777 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:52.778 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:52.778 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:30:52.778 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:53.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 177372 11.06 MB/sec execute 249 sec latency 28.182 ms 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: Reconfiguring osd.4 (monmap changed)... 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.4 on smithi099 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:30:53.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:53.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: Reconfiguring osd.4 (monmap changed)... 2024-03-11T22:30:53.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.4 on smithi099 2024-03-11T22:30:53.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:53.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:53.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:30:53.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:54.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 177838 11.09 MB/sec execute 250 sec latency 35.835 ms 2024-03-11T22:30:54.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:54 smithi073 ceph-mon[66481]: Reconfiguring osd.5 (monmap changed)... 2024-03-11T22:30:54.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:54 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.5 on smithi099 2024-03-11T22:30:54.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:54 smithi073 ceph-mon[66481]: pgmap v33: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 394 op/s 2024-03-11T22:30:54.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:54.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:54 smithi099 ceph-mon[53203]: Reconfiguring osd.5 (monmap changed)... 2024-03-11T22:30:54.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:54 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.5 on smithi099 2024-03-11T22:30:54.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:54 smithi099 ceph-mon[53203]: pgmap v33: 97 pgs: 97 active+clean; 327 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 394 op/s 2024-03-11T22:30:54.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:55.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 178277 11.07 MB/sec execute 251 sec latency 44.370 ms 2024-03-11T22:30:56.314 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: Reconfiguring osd.6 (monmap changed)... 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.6 on smithi099 2024-03-11T22:30:56.315 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:56 smithi073 ceph-mon[66481]: pgmap v34: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.8 MiB/s wr, 643 op/s 2024-03-11T22:30:56.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 178578 11.05 MB/sec execute 252 sec latency 55.738 ms 2024-03-11T22:30:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: Reconfiguring osd.6 (monmap changed)... 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.6 on smithi099 2024-03-11T22:30:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:56 smithi099 ceph-mon[53203]: pgmap v34: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.8 MiB/s wr, 643 op/s 2024-03-11T22:30:57.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 179067 11.06 MB/sec execute 253 sec latency 52.631 ms 2024-03-11T22:30:58.304 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: Reconfiguring osd.7 (monmap changed)... 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: Reconfiguring daemon osd.7 on smithi099 2024-03-11T22:30:58.305 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:30:58 smithi073 ceph-mon[66481]: pgmap v35: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-03-11T22:30:58.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 179654 11.06 MB/sec execute 254 sec latency 34.831 ms 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: Reconfiguring osd.7 (monmap changed)... 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: Reconfiguring daemon osd.7 on smithi099 2024-03-11T22:30:58.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:30:58 smithi099 ceph-mon[53203]: pgmap v35: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-03-11T22:30:59.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 180054 11.07 MB/sec execute 255 sec latency 66.338 ms 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: Reconfiguring mds.foofs.smithi099.kagazk (monmap changed)... 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: Reconfiguring daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:31:00.324 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:00 smithi073 ceph-mon[66481]: pgmap v36: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: Reconfiguring mds.foofs.smithi099.kagazk (monmap changed)... 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: Reconfiguring daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:31:00.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:00 smithi099 ceph-mon[53203]: pgmap v36: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-03-11T22:31:00.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 180647 11.09 MB/sec execute 256 sec latency 62.533 ms 2024-03-11T22:31:01.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 181089 11.12 MB/sec execute 257 sec latency 45.042 ms 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all mon 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi073"}]: dispatch 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi073"}]': finished 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi099"}]: dispatch 2024-03-11T22:31:02.118 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi099"}]': finished 2024-03-11T22:31:02.119 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:01 smithi073 ceph-mon[66481]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 731 op/s 2024-03-11T22:31:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all mon 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi073"}]: dispatch 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi073"}]': finished 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi099"}]: dispatch 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi099"}]': finished 2024-03-11T22:31:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:01 smithi099 ceph-mon[53203]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 731 op/s 2024-03-11T22:31:02.683 INFO:teuthology.orchestra.run.smithi073.stdout: 5 181479 11.12 MB/sec execute 258 sec latency 50.564 ms 2024-03-11T22:31:03.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 181884 11.10 MB/sec execute 259 sec latency 27.507 ms 2024-03-11T22:31:04.645 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: Upgrade: Updating crash.smithi073 (1/2) 2024-03-11T22:31:04.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:04.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:31:04.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:04.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: Deploying daemon crash.smithi073 on smithi073 2024-03-11T22:31:04.646 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:04 smithi073 ceph-mon[66481]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 527 op/s 2024-03-11T22:31:04.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 182211 11.09 MB/sec execute 260 sec latency 31.518 ms 2024-03-11T22:31:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: Upgrade: Updating crash.smithi073 (1/2) 2024-03-11T22:31:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi073", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:31:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: Deploying daemon crash.smithi073 on smithi073 2024-03-11T22:31:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:04 smithi099 ceph-mon[53203]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 527 op/s 2024-03-11T22:31:05.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 182748 11.11 MB/sec execute 261 sec latency 54.091 ms 2024-03-11T22:31:06.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 183244 11.10 MB/sec execute 262 sec latency 40.719 ms 2024-03-11T22:31:06.906 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:06 smithi073 ceph-mon[66481]: pgmap v39: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 791 op/s 2024-03-11T22:31:06.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:06 smithi099 ceph-mon[53203]: pgmap v39: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 791 op/s 2024-03-11T22:31:07.684 INFO:teuthology.orchestra.run.smithi073.stdout: 5 183812 11.12 MB/sec execute 263 sec latency 42.545 ms 2024-03-11T22:31:08.668 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:08 smithi073 ceph-mon[66481]: pgmap v40: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-03-11T22:31:08.690 INFO:teuthology.orchestra.run.smithi073.stdout: 5 184408 11.16 MB/sec execute 264 sec latency 43.666 ms 2024-03-11T22:31:08.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:08 smithi099 ceph-mon[53203]: pgmap v40: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-03-11T22:31:09.691 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:09.693 INFO:teuthology.orchestra.run.smithi073.stdout: 5 184836 11.17 MB/sec execute 265 sec latency 54.382 ms 2024-03-11T22:31:09.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:10.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 185104 11.14 MB/sec execute 266 sec latency 74.434 ms 2024-03-11T22:31:10.733 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:10 smithi073 ceph-mon[66481]: pgmap v41: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-03-11T22:31:10.733 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:10.733 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:10.957 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:10 smithi099 ceph-mon[53203]: pgmap v41: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-03-11T22:31:10.958 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:10.958 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:11.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 185370 11.12 MB/sec execute 267 sec latency 57.780 ms 2024-03-11T22:31:12.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 185716 11.12 MB/sec execute 268 sec latency 42.808 ms 2024-03-11T22:31:12.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:12 smithi099 ceph-mon[53203]: pgmap v42: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 856 op/s 2024-03-11T22:31:12.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:12 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:12.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:12 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:31:12.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:12 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:12.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:12 smithi073 ceph-mon[66481]: pgmap v42: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 856 op/s 2024-03-11T22:31:12.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:12 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:12.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:12 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:31:12.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:12 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:13.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 186490 11.15 MB/sec execute 269 sec latency 34.078 ms 2024-03-11T22:31:13.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:13 smithi073 ceph-mon[66481]: Upgrade: Updating crash.smithi099 (2/2) 2024-03-11T22:31:13.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:13 smithi073 ceph-mon[66481]: Deploying daemon crash.smithi099 on smithi099 2024-03-11T22:31:13.864 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:13 smithi099 ceph-mon[53203]: Upgrade: Updating crash.smithi099 (2/2) 2024-03-11T22:31:13.864 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:13 smithi099 ceph-mon[53203]: Deploying daemon crash.smithi099 on smithi099 2024-03-11T22:31:14.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 186972 11.14 MB/sec execute 270 sec latency 37.073 ms 2024-03-11T22:31:14.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:14 smithi073 ceph-mon[66481]: pgmap v43: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.2 MiB/s wr, 577 op/s 2024-03-11T22:31:14.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:14 smithi099 ceph-mon[53203]: pgmap v43: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.2 MiB/s wr, 577 op/s 2024-03-11T22:31:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:31:15.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 187283 11.13 MB/sec execute 271 sec latency 56.324 ms 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (3m) 73s ago 13m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 starting - - - - 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 starting - - - - 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (3m) 73s ago 12m 45.6M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (7m) 73s ago 7m 37.7M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (7m) 57s ago 7m 13.6M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (2m) 73s ago 14m 478M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:31:16.322 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (113s) 57s ago 12m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (93s) 73s ago 14m 30.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (64s) 57s ago 12m 24.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (6m) 73s ago 6m 279M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (6m) 57s ago 6m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (3m) 73s ago 12m 9911k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (2m) 57s ago 12m 19.9M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (11m) 73s ago 11m 558M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (10m) 73s ago 10m 479M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (10m) 73s ago 10m 357M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (10m) 73s ago 10m 498M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (9m) 57s ago 9m 528M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (9m) 57s ago 9m 336M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (8m) 57s ago 8m 539M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:31:16.323 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (8m) 57s ago 8m 446M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:31:16.324 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (119s) 73s ago 11m 55.6M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:31:16.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 187601 11.13 MB/sec execute 272 sec latency 49.138 ms 2024-03-11T22:31:16.834 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:16 smithi073 ceph-mon[66481]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:16.835 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:16 smithi073 ceph-mon[66481]: pgmap v44: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 844 op/s 2024-03-11T22:31:16.835 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:16.835 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:16.835 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:16.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:16 smithi099 ceph-mon[53203]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:16.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:16 smithi099 ceph-mon[53203]: pgmap v44: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 844 op/s 2024-03-11T22:31:16.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:16.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:16.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:31:17.000 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:31:17.001 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr" 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:31:17.557 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "4/23 daemons upgraded", 2024-03-11T22:31:17.558 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading crash daemons", 2024-03-11T22:31:17.558 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:31:17.558 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:31:17.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 187837 11.12 MB/sec execute 273 sec latency 53.898 ms 2024-03-11T22:31:17.848 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:17 smithi073 ceph-mon[66481]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:17.848 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:17 smithi073 ceph-mon[66481]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:17.848 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:17 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2916078841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:17.958 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:17 smithi099 ceph-mon[53203]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:17.958 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:17 smithi099 ceph-mon[53203]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:17.958 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:17 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2916078841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:18.194 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:31:18.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 188335 11.15 MB/sec execute 274 sec latency 55.149 ms 2024-03-11T22:31:18.717 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:18 smithi073 ceph-mon[66481]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:18.717 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:18 smithi073 ceph-mon[66481]: pgmap v45: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 581 op/s 2024-03-11T22:31:18.717 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:18 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/368711743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:31:18.931 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:18 smithi099 ceph-mon[53203]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:18.931 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:18 smithi099 ceph-mon[53203]: pgmap v45: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 581 op/s 2024-03-11T22:31:18.931 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:18 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/368711743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:31:19.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 188801 11.14 MB/sec execute 275 sec latency 68.566 ms 2024-03-11T22:31:19.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:19 smithi073 ceph-mon[66481]: pgmap v46: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 581 op/s 2024-03-11T22:31:19.975 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:19 smithi099 ceph-mon[53203]: pgmap v46: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 581 op/s 2024-03-11T22:31:20.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 189160 11.12 MB/sec execute 276 sec latency 40.540 ms 2024-03-11T22:31:21.650 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:21.651 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:21.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 189530 11.12 MB/sec execute 277 sec latency 40.370 ms 2024-03-11T22:31:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:21.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:22.664 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:22 smithi073 ceph-mon[66481]: pgmap v47: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-03-11T22:31:22.664 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:22.665 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:22.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 190047 11.13 MB/sec execute 278 sec latency 36.754 ms 2024-03-11T22:31:22.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:22 smithi099 ceph-mon[53203]: pgmap v47: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 795 op/s 2024-03-11T22:31:22.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:22.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:23.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 190497 11.12 MB/sec execute 279 sec latency 30.930 ms 2024-03-11T22:31:24.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 190940 11.13 MB/sec execute 280 sec latency 37.020 ms 2024-03-11T22:31:24.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:24 smithi073 ceph-mon[66481]: pgmap v48: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 482 op/s 2024-03-11T22:31:24.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:24 smithi099 ceph-mon[53203]: pgmap v48: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 482 op/s 2024-03-11T22:31:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:25.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 191356 11.14 MB/sec execute 281 sec latency 53.638 ms 2024-03-11T22:31:25.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:25 smithi073 ceph-mon[66481]: mgrmap e41: smithi073.pglkvm(active, since 91s), standbys: smithi099.rcoixq 2024-03-11T22:31:25.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:25.716 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:25.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:25 smithi099 ceph-mon[53203]: mgrmap e41: smithi073.pglkvm(active, since 91s), standbys: smithi099.rcoixq 2024-03-11T22:31:25.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:25.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 191590 11.13 MB/sec execute 282 sec latency 55.689 ms 2024-03-11T22:31:26.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:26 smithi073 ceph-mon[66481]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 757 op/s 2024-03-11T22:31:26.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:26 smithi099 ceph-mon[53203]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 757 op/s 2024-03-11T22:31:26.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:26.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 191859 11.13 MB/sec execute 283 sec latency 80.157 ms 2024-03-11T22:31:27.789 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:27 smithi073 ceph-mon[66481]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-03-11T22:31:27.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:27 smithi099 ceph-mon[53203]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-03-11T22:31:28.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 192091 11.11 MB/sec execute 284 sec latency 60.180 ms 2024-03-11T22:31:29.695 INFO:teuthology.orchestra.run.smithi073.stdout: 5 192359 11.09 MB/sec execute 285 sec latency 63.803 ms 2024-03-11T22:31:30.695 INFO:teuthology.orchestra.run.smithi073.stdout: 5 192593 11.06 MB/sec execute 286 sec latency 61.964 ms 2024-03-11T22:31:30.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:30 smithi073 ceph-mon[66481]: pgmap v51: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-03-11T22:31:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:30 smithi099 ceph-mon[53203]: pgmap v51: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.9 MiB/s wr, 490 op/s 2024-03-11T22:31:31.695 INFO:teuthology.orchestra.run.smithi073.stdout: 5 192836 11.05 MB/sec execute 287 sec latency 54.321 ms 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: pgmap v52: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 659 op/s 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:31:32.671 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi073"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi073"}]': finished 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi099"}]: dispatch 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi099"}]': finished 2024-03-11T22:31:32.672 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T22:31:32.694 INFO:teuthology.orchestra.run.smithi073.stdout: 5 193090 11.03 MB/sec execute 288 sec latency 54.126 ms 2024-03-11T22:31:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: pgmap v52: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 659 op/s 2024-03-11T22:31:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi073"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi073"}]': finished 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi099"}]: dispatch 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi099"}]': finished 2024-03-11T22:31:32.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T22:31:33.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:33 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all crash 2024-03-11T22:31:33.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:33 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T22:31:33.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:33 smithi073 ceph-mon[66481]: Upgrade: osd.0 is safe to restart 2024-03-11T22:31:33.700 INFO:teuthology.orchestra.run.smithi073.stdout: 5 193442 11.02 MB/sec execute 289 sec latency 60.066 ms 2024-03-11T22:31:33.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:33 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all crash 2024-03-11T22:31:33.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:33 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-11T22:31:33.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:33 smithi099 ceph-mon[53203]: Upgrade: osd.0 is safe to restart 2024-03-11T22:31:34.692 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:34 smithi073 ceph-mon[66481]: pgmap v53: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-03-11T22:31:34.707 INFO:teuthology.orchestra.run.smithi073.stdout: 5 193919 11.03 MB/sec execute 290 sec latency 39.809 ms 2024-03-11T22:31:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:34 smithi099 ceph-mon[53203]: pgmap v53: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-03-11T22:31:35.709 INFO:teuthology.orchestra.run.smithi073.stdout: 5 194284 11.01 MB/sec execute 291 sec latency 40.129 ms 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: Upgrade: Updating osd.0 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: Deploying daemon osd.0 on smithi073 2024-03-11T22:31:35.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:35 smithi073 ceph-mon[66481]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 647 op/s 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: Upgrade: Updating osd.0 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: Deploying daemon osd.0 on smithi073 2024-03-11T22:31:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:35 smithi099 ceph-mon[53203]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 647 op/s 2024-03-11T22:31:36.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 194675 11.01 MB/sec execute 292 sec latency 31.484 ms 2024-03-11T22:31:37.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 195061 11.03 MB/sec execute 293 sec latency 46.716 ms 2024-03-11T22:31:38.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 195385 11.04 MB/sec execute 294 sec latency 44.807 ms 2024-03-11T22:31:38.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:38 smithi073 ceph-mon[66481]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s 2024-03-11T22:31:38.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:38 smithi099 ceph-mon[53203]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s 2024-03-11T22:31:39.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 195664 11.02 MB/sec execute 295 sec latency 55.183 ms 2024-03-11T22:31:39.830 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:39.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:40.088 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:39 smithi073 systemd[1]: Stopping Ceph osd.0 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:31:40.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:40 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[35206]: 2024-03-11T22:31:40.328+0000 7f3292810700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:31:40.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:40 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[35206]: 2024-03-11T22:31:40.328+0000 7f3292810700 -1 osd.0 74 *** Got signal Terminated *** 2024-03-11T22:31:40.339 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:40 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[35206]: 2024-03-11T22:31:40.328+0000 7f3292810700 -1 osd.0 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:31:40.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 196047 11.01 MB/sec execute 296 sec latency 44.946 ms 2024-03-11T22:31:40.845 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:40 smithi073 ceph-mon[66481]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.9 MiB/s wr, 480 op/s 2024-03-11T22:31:40.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:40 smithi073 ceph-mon[66481]: osd.0 marked itself down and dead 2024-03-11T22:31:40.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:40 smithi099 ceph-mon[53203]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.9 MiB/s wr, 480 op/s 2024-03-11T22:31:40.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:40 smithi099 ceph-mon[53203]: osd.0 marked itself down and dead 2024-03-11T22:31:41.352 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:41 smithi073 podman[72018]: 2024-03-11 22:31:41.330085141 +0000 UTC m=+1.136945953 container died 85b504bcbb223ef8d501fd8fbd4eee5e169912b05c0c854c23d3c20832c64c18 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.openshift.expose-services=, vcs-type=git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, RELEASE=HEAD) 2024-03-11T22:31:41.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 196324 10.98 MB/sec execute 297 sec latency 736.724 ms 2024-03-11T22:31:41.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:41 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:31:41.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:41 smithi073 ceph-mon[66481]: failed to encode map e75 with expected crc 2024-03-11T22:31:41.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:41 smithi073 ceph-mon[66481]: osdmap e75: 8 total, 7 up, 8 in 2024-03-11T22:31:41.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:41 smithi073 ceph-mon[66481]: failed to encode map e75 with expected crc 2024-03-11T22:31:41.863 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:41 smithi073 ceph-mon[66481]: pgmap v58: 97 pgs: 13 stale+active+clean, 84 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 498 op/s 2024-03-11T22:31:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:41 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:31:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:41 smithi099 ceph-mon[53203]: failed to encode map e75 with expected crc 2024-03-11T22:31:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:41 smithi099 ceph-mon[53203]: osdmap e75: 8 total, 7 up, 8 in 2024-03-11T22:31:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:41 smithi099 ceph-mon[53203]: failed to encode map e75 with expected crc 2024-03-11T22:31:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:41 smithi099 ceph-mon[53203]: pgmap v58: 97 pgs: 13 stale+active+clean, 84 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 498 op/s 2024-03-11T22:31:42.368 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:42 smithi073 podman[72018]: 2024-03-11 22:31:42.283062378 +0000 UTC m=+2.089923187 container cleanup 85b504bcbb223ef8d501fd8fbd4eee5e169912b05c0c854c23d3c20832c64c18 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-type=git, RELEASE=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, ceph=True, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-11T22:31:42.368 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:42 smithi073 bash[72018]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:31:42.618 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:42 smithi073 podman[72034]: 2024-03-11 22:31:42.598846584 +0000 UTC m=+1.269888542 container remove 85b504bcbb223ef8d501fd8fbd4eee5e169912b05c0c854c23d3c20832c64c18 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.expose-services=, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, ceph=True, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T22:31:42.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 196693 10.98 MB/sec execute 298 sec latency 772.870 ms 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: osdmap e76: 8 total, 7 up, 8 in 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:42 smithi073 ceph-mon[66481]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: osdmap e76: 8 total, 7 up, 8 in 2024-03-11T22:31:42.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:42.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:42 smithi099 ceph-mon[53203]: failed to encode map e76 with expected crc 2024-03-11T22:31:43.121 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:43 smithi073 podman[72117]: 2024-03-11 22:31:42.975937523 +0000 UTC m=+0.039940702 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:31:43.624 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:43 smithi073 podman[72117]: 2024-03-11 22:31:43.490729773 +0000 UTC m=+0.554732952 container create ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:31:43.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 197273 10.99 MB/sec execute 299 sec latency 38.421 ms 2024-03-11T22:31:43.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:43 smithi099 ceph-mon[53203]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 318 op/s 2024-03-11T22:31:43.873 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:43 smithi073 ceph-mon[66481]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 318 op/s 2024-03-11T22:31:44.126 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:43 smithi073 podman[72117]: 2024-03-11 22:31:43.893738766 +0000 UTC m=+0.957741964 container init ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:31:44.126 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:43 smithi073 podman[72117]: 2024-03-11 22:31:43.905689002 +0000 UTC m=+0.969692193 container start ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T22:31:44.126 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:44 smithi073 podman[72117]: 2024-03-11 22:31:44.057749286 +0000 UTC m=+1.121752470 container attach ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T22:31:44.379 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:44 smithi073 podman[72117]: 2024-03-11 22:31:44.369073304 +0000 UTC m=+1.433076482 container died ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T22:31:44.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 197799 11.00 MB/sec execute 300 sec latency 29.329 ms 2024-03-11T22:31:45.646 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:45 smithi073 podman[72143]: 2024-03-11 22:31:45.465098702 +0000 UTC m=+1.089422980 container remove ec414a723fd707d5f36fe26d422b45edf8fe34e05793e0ab3d6f28477ef9dfe4 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T22:31:45.646 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:45 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-11T22:31:45.646 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:45 smithi073 systemd[1]: Stopped Ceph osd.0 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:31:45.646 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:45 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.0.service: Consumed 3min 20.924s CPU time. 2024-03-11T22:31:45.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 198223 11.00 MB/sec execute 301 sec latency 41.818 ms 2024-03-11T22:31:45.914 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:45 smithi073 systemd[1]: Starting Ceph osd.0 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:31:45.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:45 smithi073 ceph-mon[66481]: pgmap v61: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.9 MiB/s wr, 706 op/s; 222/1683 objects degraded (13.191%) 2024-03-11T22:31:45.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:45 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 222/1683 objects degraded (13.191%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:45.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:45 smithi099 ceph-mon[53203]: pgmap v61: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.9 MiB/s wr, 706 op/s; 222/1683 objects degraded (13.191%) 2024-03-11T22:31:45.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:45 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 222/1683 objects degraded (13.191%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:46.168 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:46 smithi073 podman[72246]: 2024-03-11 22:31:46.027069423 +0000 UTC m=+0.033737727 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:31:46.686 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:46 smithi073 podman[72246]: 2024-03-11 22:31:46.541943663 +0000 UTC m=+0.548611956 container create 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T22:31:46.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 198658 11.03 MB/sec execute 302 sec latency 65.513 ms 2024-03-11T22:31:47.188 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:46 smithi073 podman[72246]: 2024-03-11 22:31:46.974416529 +0000 UTC m=+0.981084831 container init 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T22:31:47.188 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:46 smithi073 podman[72246]: 2024-03-11 22:31:46.986096533 +0000 UTC m=+0.992764834 container start 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:31:47.447 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 podman[72246]: 2024-03-11 22:31:47.431471485 +0000 UTC m=+1.438139784 container attach 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T22:31:47.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T22:31:47.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T22:31:47.699 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 bash[72246]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate[72292]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-11T22:31:47.700 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:47 smithi073 podman[72246]: 2024-03-11 22:31:47.628281703 +0000 UTC m=+1.634950002 container died 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T22:31:47.711 INFO:teuthology.orchestra.run.smithi073.stdout: 5 199132 11.05 MB/sec execute 303 sec latency 32.561 ms 2024-03-11T22:31:48.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 199465 11.02 MB/sec execute 304 sec latency 40.129 ms 2024-03-11T22:31:48.742 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:48 smithi073 ceph-mon[66481]: pgmap v62: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.1 MiB/s wr, 544 op/s; 222/1683 objects degraded (13.191%) 2024-03-11T22:31:48.781 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:31:48.894 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:48 smithi073 podman[72424]: 2024-03-11 22:31:48.788491478 +0000 UTC m=+1.154764886 container remove 803bf3e8fd4a3ef2f54c216c9a50180cfceba375a4acbf2420e90b4e1228722a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T22:31:48.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:48 smithi099 ceph-mon[53203]: pgmap v62: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.1 MiB/s wr, 544 op/s; 222/1683 objects degraded (13.191%) 2024-03-11T22:31:49.262 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 podman[72525]: 2024-03-11 22:31:48.957854049 +0000 UTC m=+0.030756071 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:31:49.262 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 podman[72525]: 2024-03-11 22:31:49.121609235 +0000 UTC m=+0.194511238 container create c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T22:31:49.513 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 podman[72525]: 2024-03-11 22:31:49.507273825 +0000 UTC m=+0.580175827 container init c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (4m) 24s ago 13m 17.4M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (40s) 24s ago 13m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (34s) 29s ago 12m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (4m) 24s ago 13m 44.5M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (7m) 24s ago 7m 37.8M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (7m) 29s ago 7m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (3m) 24s ago 14m 497M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (2m) 29s ago 12m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (2m) 24s ago 14m 43.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (97s) 29s ago 12m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (7m) 24s ago 7m 275M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (7m) 29s ago 7m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:31:49.688 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (3m) 24s ago 13m 11.9M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (3m) 29s ago 12m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (12m) 24s ago 12m 387M 1936M 17.2.0 e1d6a67b021e 85b504bcbb22 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (11m) 24s ago 11m 576M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (11m) 24s ago 11m 461M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (10m) 24s ago 10m 591M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (10m) 29s ago 10m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (9m) 29s ago 9m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (9m) 29s ago 9m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (9m) 29s ago 9m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:31:49.689 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (2m) 24s ago 12m 58.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:31:49.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 199762 11.00 MB/sec execute 305 sec latency 30.246 ms 2024-03-11T22:31:49.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 podman[72525]: 2024-03-11 22:31:49.518702638 +0000 UTC m=+0.591604641 container start c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T22:31:49.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 ceph-osd[72567]: -- 172.21.15.73:0/4278115192 <== mon.1 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570275e71e0 con 0x5570275b8c00 2024-03-11T22:31:49.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 bash[72525]: c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 2024-03-11T22:31:49.766 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:49 smithi073 systemd[1]: Started Ceph osd.0 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:31:50.272 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:31:50.096+0000 7f36c6bb27c0 -1 Falling back to public interface 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:31:50.285 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:31:50.286 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-11T22:31:50.286 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T22:31:50.286 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:31:50.286 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:31:50.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 200079 11.00 MB/sec execute 306 sec latency 67.472 ms 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:31:50.769 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "6/23 daemons upgraded", 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:31:50.770 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: pgmap v63: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.6 MiB/s wr, 596 op/s; 228/1728 objects degraded (13.194%) 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:50.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:50 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/3048774440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: pgmap v63: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 343 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.6 MiB/s wr, 596 op/s; 228/1728 objects degraded (13.194%) 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:50 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/3048774440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:31:51.354 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 228/1728 objects degraded (13.194%), 28 pgs degraded 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: osd.0 (root=default,host=smithi073) is down 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 228/1728 objects degraded (13.194%), 28 pgs degraded 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.8 is active+undersized+degraded, acting [2,1] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-03-11T22:31:51.355 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-11T22:31:51.356 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-11T22:31:51.357 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-03-11T22:31:51.357 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-03-11T22:31:51.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 200414 10.99 MB/sec execute 307 sec latency 59.996 ms 2024-03-11T22:31:51.829 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:51 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/620076149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:31:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:51 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/620076149' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:31:52.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 200817 10.98 MB/sec execute 308 sec latency 35.616 ms 2024-03-11T22:31:52.890 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:52 smithi073 ceph-mon[66481]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:52.891 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:52 smithi073 ceph-mon[66481]: pgmap v64: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.0 MiB/s wr, 604 op/s; 219/1722 objects degraded (12.718%) 2024-03-11T22:31:52.891 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:52 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 219/1722 objects degraded (12.718%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:52.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:52 smithi099 ceph-mon[53203]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:31:52.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:52 smithi099 ceph-mon[53203]: pgmap v64: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.0 MiB/s wr, 604 op/s; 219/1722 objects degraded (12.718%) 2024-03-11T22:31:52.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:52 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 219/1722 objects degraded (12.718%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:53.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 201135 10.97 MB/sec execute 309 sec latency 58.805 ms 2024-03-11T22:31:54.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 201475 10.96 MB/sec execute 310 sec latency 45.013 ms 2024-03-11T22:31:54.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:54 smithi073 ceph-mon[66481]: pgmap v65: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 507 op/s; 219/1722 objects degraded (12.718%) 2024-03-11T22:31:54.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:54.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:54.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:54 smithi099 ceph-mon[53203]: pgmap v65: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 348 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.1 MiB/s wr, 507 op/s; 219/1722 objects degraded (12.718%) 2024-03-11T22:31:54.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:54.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:55.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 201749 10.95 MB/sec execute 311 sec latency 46.873 ms 2024-03-11T22:31:56.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 202050 10.96 MB/sec execute 312 sec latency 46.630 ms 2024-03-11T22:31:56.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:56 smithi073 ceph-mon[66481]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 339 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 718 op/s; 206/1617 objects degraded (12.740%) 2024-03-11T22:31:56.868 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:56 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:31:56.712+0000 7f36c6bb27c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-11T22:31:56.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:56 smithi099 ceph-mon[53203]: pgmap v66: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 339 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 718 op/s; 206/1617 objects degraded (12.740%) 2024-03-11T22:31:57.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 202343 10.96 MB/sec execute 313 sec latency 63.342 ms 2024-03-11T22:31:58.136 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:57 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:31:57.892+0000 7f36c6bb27c0 -1 osd.0 74 log_to_monitors true 2024-03-11T22:31:58.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 202744 10.97 MB/sec execute 314 sec latency 37.328 ms 2024-03-11T22:31:58.896 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:31:58.729+0000 7f36b800a700 -1 osd.0 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:31:58.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-mon[66481]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 339 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 459 op/s; 206/1617 objects degraded (12.740%) 2024-03-11T22:31:58.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 206/1617 objects degraded (12.740%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:58.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-mon[66481]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:31:58.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:58.896 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:58 smithi099 ceph-mon[53203]: pgmap v67: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 339 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 459 op/s; 206/1617 objects degraded (12.740%) 2024-03-11T22:31:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:58 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 206/1617 objects degraded (12.740%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:31:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:58 smithi099 ceph-mon[53203]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:31:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:31:59.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 203141 10.95 MB/sec execute 315 sec latency 44.244 ms 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: osdmap e77: 8 total, 7 up, 8 in 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.912 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:31:59 smithi073 ceph-mon[66481]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 350 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 532 op/s; 215/1677 objects degraded (12.821%) 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: osdmap e77: 8 total, 7 up, 8 in 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: from='osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: failed to encode map e77 with expected crc 2024-03-11T22:31:59.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:31:59 smithi099 ceph-mon[53203]: pgmap v69: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 350 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 532 op/s; 215/1677 objects degraded (12.821%) 2024-03-11T22:32:00.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 203311 10.92 MB/sec execute 316 sec latency 1000.215 ms 2024-03-11T22:32:00.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:32:00.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: failed to encode map e78 with expected crc 2024-03-11T22:32:00.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667] boot 2024-03-11T22:32:00.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T22:32:00.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:32:00.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:00.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: failed to encode map e78 with expected crc 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: osd.0 [v2:172.21.15.73:6802/1561151667,v1:172.21.15.73:6803/1561151667] boot 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:00.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:01.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 203335 10.89 MB/sec execute 317 sec latency 2000.247 ms 2024-03-11T22:32:01.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:01 smithi073 ceph-mon[66481]: pgmap v72: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.6 MiB/s wr, 401 op/s; 235/1713 objects degraded (13.719%) 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: failed to encode map e79 with expected crc 2024-03-11T22:32:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:01 smithi099 ceph-mon[53203]: pgmap v72: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.6 MiB/s wr, 401 op/s; 235/1713 objects degraded (13.719%) 2024-03-11T22:32:02.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 203595 10.88 MB/sec execute 318 sec latency 2327.168 ms 2024-03-11T22:32:02.786 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.786 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T22:32:02.786 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.786 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.787 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:02 smithi073 ceph-mon[66481]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:02.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:02 smithi099 ceph-mon[53203]: failed to encode map e80 with expected crc 2024-03-11T22:32:03.712 INFO:teuthology.orchestra.run.smithi073.stdout: 5 203851 10.87 MB/sec execute 319 sec latency 956.939 ms 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 235/1713 objects degraded (13.719%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:03.812 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:03 smithi073 ceph-mon[66481]: pgmap v75: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 1.7 MiB/s wr, 91 op/s; 235/1713 objects degraded (13.719%) 2024-03-11T22:32:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T22:32:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: failed to encode map e81 with expected crc 2024-03-11T22:32:03.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 235/1713 objects degraded (13.719%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:03.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:03 smithi099 ceph-mon[53203]: pgmap v75: 97 pgs: 28 active+undersized+degraded, 8 active+undersized, 61 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 1.7 MiB/s wr, 91 op/s; 235/1713 objects degraded (13.719%) 2024-03-11T22:32:04.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 204295 10.88 MB/sec execute 320 sec latency 1007.432 ms 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:04 smithi073 ceph-mon[66481]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T22:32:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:04 smithi099 ceph-mon[53203]: failed to encode map e82 with expected crc 2024-03-11T22:32:05.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 204549 10.85 MB/sec execute 321 sec latency 934.328 ms 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: failed to encode map e83 with expected crc 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:05.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-11T22:32:05.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:05 smithi073 ceph-mon[66481]: pgmap v78: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 446 op/s; 155/1650 objects degraded (9.394%); 2.0 MiB/s, 24 objects/s recovering 2024-03-11T22:32:06.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: failed to encode map e83 with expected crc 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:05 smithi099 ceph-mon[53203]: pgmap v78: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 446 op/s; 155/1650 objects degraded (9.394%); 2.0 MiB/s, 24 objects/s recovering 2024-03-11T22:32:06.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 204894 10.83 MB/sec execute 322 sec latency 1001.172 ms 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.928 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:06.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:06 smithi073 ceph-mon[66481]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T22:32:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:06 smithi099 ceph-mon[53203]: failed to encode map e84 with expected crc 2024-03-11T22:32:07.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 205179 10.82 MB/sec execute 323 sec latency 943.590 ms 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: failed to encode map e85 with expected crc 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: failed to encode map e85 with expected crc 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: failed to encode map e85 with expected crc 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: failed to encode map e85 with expected crc 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: failed to encode map e85 with expected crc 2024-03-11T22:32:07.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:07 smithi073 ceph-mon[66481]: pgmap v81: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 446 op/s; 155/1650 objects degraded (9.394%); 2.0 MiB/s, 24 objects/s recovering 2024-03-11T22:32:08.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: failed to encode map e85 with expected crc 2024-03-11T22:32:08.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T22:32:08.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: failed to encode map e85 with expected crc 2024-03-11T22:32:08.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: failed to encode map e85 with expected crc 2024-03-11T22:32:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: failed to encode map e85 with expected crc 2024-03-11T22:32:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: failed to encode map e85 with expected crc 2024-03-11T22:32:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:07 smithi099 ceph-mon[53203]: pgmap v81: 97 pgs: 86 active+clean, 2 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 446 op/s; 155/1650 objects degraded (9.394%); 2.0 MiB/s, 24 objects/s recovering 2024-03-11T22:32:08.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 205653 10.84 MB/sec execute 324 sec latency 556.288 ms 2024-03-11T22:32:08.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: failed to encode map e86 with expected crc 2024-03-11T22:32:08.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T22:32:08.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: failed to encode map e86 with expected crc 2024-03-11T22:32:08.968 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: failed to encode map e86 with expected crc 2024-03-11T22:32:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 155/1650 objects degraded (9.394%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:09.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: failed to encode map e86 with expected crc 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: failed to encode map e86 with expected crc 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: failed to encode map e86 with expected crc 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 155/1650 objects degraded (9.394%), 7 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:09.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 205953 10.84 MB/sec execute 325 sec latency 567.487 ms 2024-03-11T22:32:09.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:09 smithi073 ceph-mon[66481]: failed to encode map e87 with expected crc 2024-03-11T22:32:09.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:09 smithi073 ceph-mon[66481]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T22:32:09.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:09 smithi073 ceph-mon[66481]: failed to encode map e87 with expected crc 2024-03-11T22:32:09.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:09 smithi073 ceph-mon[66481]: pgmap v84: 97 pgs: 94 active+clean, 3 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 145 op/s; 88/1641 objects degraded (5.363%); 1024 KiB/s, 28 objects/s recovering 2024-03-11T22:32:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:09 smithi099 ceph-mon[53203]: failed to encode map e87 with expected crc 2024-03-11T22:32:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:09 smithi099 ceph-mon[53203]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T22:32:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:09 smithi099 ceph-mon[53203]: failed to encode map e87 with expected crc 2024-03-11T22:32:10.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:09 smithi099 ceph-mon[53203]: pgmap v84: 97 pgs: 94 active+clean, 3 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 145 op/s; 88/1641 objects degraded (5.363%); 1024 KiB/s, 28 objects/s recovering 2024-03-11T22:32:10.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 206452 10.88 MB/sec execute 326 sec latency 660.708 ms 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: failed to encode map e88 with expected crc 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: failed to encode map e88 with expected crc 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: failed to encode map e88 with expected crc 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: failed to encode map e88 with expected crc 2024-03-11T22:32:10.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:10 smithi073 ceph-mon[66481]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T22:32:11.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:10 smithi099 ceph-mon[53203]: failed to encode map e88 with expected crc 2024-03-11T22:32:11.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 206760 10.87 MB/sec execute 327 sec latency 34.856 ms 2024-03-11T22:32:11.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:11 smithi073 ceph-mon[66481]: pgmap v86: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 508 op/s; 29/1635 objects degraded (1.774%); 843 KiB/s, 35 objects/s recovering 2024-03-11T22:32:12.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:11 smithi099 ceph-mon[53203]: pgmap v86: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 508 op/s; 29/1635 objects degraded (1.774%); 843 KiB/s, 35 objects/s recovering 2024-03-11T22:32:12.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 207075 10.84 MB/sec execute 328 sec latency 37.228 ms 2024-03-11T22:32:13.142 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:12 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 29/1635 objects degraded (1.774%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:32:13.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:12 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 29/1635 objects degraded (1.774%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:32:13.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 207380 10.83 MB/sec execute 329 sec latency 36.917 ms 2024-03-11T22:32:14.192 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:13 smithi073 ceph-mon[66481]: pgmap v87: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 29/1635 objects degraded (1.774%); 683 KiB/s, 28 objects/s recovering 2024-03-11T22:32:14.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:13 smithi099 ceph-mon[53203]: pgmap v87: 97 pgs: 1 peering, 95 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 29/1635 objects degraded (1.774%); 683 KiB/s, 28 objects/s recovering 2024-03-11T22:32:14.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 207777 10.83 MB/sec execute 330 sec latency 44.269 ms 2024-03-11T22:32:15.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 208242 10.82 MB/sec execute 331 sec latency 39.380 ms 2024-03-11T22:32:15.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:15 smithi073 ceph-mon[66481]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 MiB/s wr, 673 op/s; 522 KiB/s, 25 objects/s recovering 2024-03-11T22:32:15.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:15 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/1635 objects degraded (1.774%), 1 pg degraded) 2024-03-11T22:32:15.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:15 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:32:15.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:15 smithi099 ceph-mon[53203]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 MiB/s wr, 673 op/s; 522 KiB/s, 25 objects/s recovering 2024-03-11T22:32:15.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:15 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/1635 objects degraded (1.774%), 1 pg degraded) 2024-03-11T22:32:15.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:15 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:32:16.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 208680 10.81 MB/sec execute 332 sec latency 33.545 ms 2024-03-11T22:32:17.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 209169 10.83 MB/sec execute 333 sec latency 31.282 ms 2024-03-11T22:32:18.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 209509 10.84 MB/sec execute 334 sec latency 54.223 ms 2024-03-11T22:32:18.779 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:18 smithi073 ceph-mon[66481]: pgmap v89: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 575 op/s; 0 B/s, 14 objects/s recovering 2024-03-11T22:32:18.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:18 smithi099 ceph-mon[53203]: pgmap v89: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.8 MiB/s wr, 575 op/s; 0 B/s, 14 objects/s recovering 2024-03-11T22:32:19.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 209884 10.85 MB/sec execute 335 sec latency 63.959 ms 2024-03-11T22:32:20.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 210091 10.84 MB/sec execute 336 sec latency 62.181 ms 2024-03-11T22:32:20.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:20 smithi073 ceph-mon[66481]: pgmap v90: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.8 MiB/s wr, 541 op/s; 0 B/s, 8 objects/s recovering 2024-03-11T22:32:20.815 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:20 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:20.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:20 smithi099 ceph-mon[53203]: pgmap v90: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.8 MiB/s wr, 541 op/s; 0 B/s, 8 objects/s recovering 2024-03-11T22:32:20.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:20 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:21.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 210264 10.83 MB/sec execute 337 sec latency 85.147 ms 2024-03-11T22:32:21.784 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:32:21.845 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:21 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:21.845 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:21 smithi073 ceph-mon[66481]: Upgrade: osd.1 is safe to restart 2024-03-11T22:32:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:21 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-11T22:32:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:21 smithi099 ceph-mon[53203]: Upgrade: osd.1 is safe to restart 2024-03-11T22:32:22.713 INFO:teuthology.orchestra.run.smithi073.stdout: 5 210456 10.80 MB/sec execute 338 sec latency 118.033 ms 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (5m) 24s ago 14m 15.7M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (74s) 24s ago 14m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (67s) 62s ago 13m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (4m) 24s ago 13m 43.8M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (8m) 24s ago 8m 38.1M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (8m) 62s ago 8m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (3m) 24s ago 15m 499M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (2m) 62s ago 13m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (2m) 24s ago 15m 46.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (2m) 62s ago 13m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:32:22.762 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (7m) 24s ago 7m 276M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (7m) 62s ago 7m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (4m) 24s ago 13m 11.6M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (4m) 62s ago 13m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (33s) 24s ago 12m 15.0M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (12m) 24s ago 12m 341M 1936M 17.2.0 e1d6a67b021e 05b928175b2b 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (11m) 24s ago 11m 501M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (11m) 24s ago 11m 360M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (10m) 62s ago 10m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (10m) 62s ago 10m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (10m) 62s ago 10m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (9m) 62s ago 9m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:32:22.763 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (3m) 24s ago 12m 60.7M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:32:22.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:22 smithi073 ceph-mon[66481]: pgmap v91: 97 pgs: 97 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:22.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:22.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:32:22.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:22.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:22 smithi099 ceph-mon[53203]: pgmap v91: 97 pgs: 97 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:22.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:22.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:32:22.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:23.341 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:32:23.342 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:32:23.343 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:32:23.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 210660 10.77 MB/sec execute 339 sec latency 60.922 ms 2024-03-11T22:32:23.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:23 smithi073 ceph-mon[66481]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:23.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:23 smithi073 ceph-mon[66481]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:23.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:23 smithi073 ceph-mon[66481]: Upgrade: Updating osd.1 2024-03-11T22:32:23.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:23 smithi073 ceph-mon[66481]: Deploying daemon osd.1 on smithi073 2024-03-11T22:32:23.868 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:23 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2017061071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:23.879 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:32:23.879 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:32:23.879 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:32:23.879 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "7/23 daemons upgraded", 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:32:23.880 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:32:23.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:23 smithi099 ceph-mon[53203]: from='client.44163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:23.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:23 smithi099 ceph-mon[53203]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:23 smithi099 ceph-mon[53203]: Upgrade: Updating osd.1 2024-03-11T22:32:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:23 smithi099 ceph-mon[53203]: Deploying daemon osd.1 on smithi073 2024-03-11T22:32:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:23 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2017061071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:24.455 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:32:24.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 210860 10.75 MB/sec execute 340 sec latency 44.910 ms 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: pgmap v92: 97 pgs: 97 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 474 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:24.901 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:24 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1721426907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: pgmap v92: 97 pgs: 97 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 474 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='client.44177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:24.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:24 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1721426907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:32:25.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 211093 10.73 MB/sec execute 341 sec latency 51.108 ms 2024-03-11T22:32:25.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:25 smithi073 ceph-mon[66481]: pgmap v93: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.3 MiB/s wr, 613 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:25.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:25 smithi099 ceph-mon[53203]: pgmap v93: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.3 MiB/s wr, 613 op/s; 0 B/s, 2 objects/s recovering 2024-03-11T22:32:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 211473 10.74 MB/sec execute 342 sec latency 73.155 ms 2024-03-11T22:32:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 211947 10.72 MB/sec execute 343 sec latency 46.283 ms 2024-03-11T22:32:28.239 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:28 smithi073 systemd[1]: Stopping Ceph osd.1 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:32:28.495 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:28 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[39266]: 2024-03-11T22:32:28.408+0000 7fe0fd6c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:32:28.495 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:28 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[39266]: 2024-03-11T22:32:28.408+0000 7fe0fd6c1700 -1 osd.1 88 *** Got signal Terminated *** 2024-03-11T22:32:28.495 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:28 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[39266]: 2024-03-11T22:32:28.408+0000 7fe0fd6c1700 -1 osd.1 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:32:28.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 212252 10.71 MB/sec execute 344 sec latency 36.338 ms 2024-03-11T22:32:28.751 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:28 smithi073 ceph-mon[66481]: pgmap v94: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 378 op/s 2024-03-11T22:32:28.751 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:28 smithi073 ceph-mon[66481]: osd.1 marked itself down and dead 2024-03-11T22:32:28.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:28 smithi099 ceph-mon[53203]: pgmap v94: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 378 op/s 2024-03-11T22:32:28.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:28 smithi099 ceph-mon[53203]: osd.1 marked itself down and dead 2024-03-11T22:32:29.416 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:29 smithi073 podman[74768]: 2024-03-11 22:32:29.151204778 +0000 UTC m=+0.932378575 container died 05b928175b2b93b3fd045e7925872b7befed2435d890a2d754c884b643b10e05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_BRANCH=HEAD) 2024-03-11T22:32:29.684 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:29 smithi073 podman[74768]: 2024-03-11 22:32:29.518507807 +0000 UTC m=+1.299681607 container cleanup 05b928175b2b93b3fd045e7925872b7befed2435d890a2d754c884b643b10e05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, build-date=2022-05-03T08:36:31.336870, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T22:32:29.685 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:29 smithi073 bash[74768]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:32:29.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 212319 10.69 MB/sec execute 345 sec latency 1004.665 ms 2024-03-11T22:32:29.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:29 smithi073 podman[74785]: 2024-03-11 22:32:29.828702081 +0000 UTC m=+0.672733934 container remove 05b928175b2b93b3fd045e7925872b7befed2435d890a2d754c884b643b10e05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-11T22:32:29.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:29 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:32:29.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:29 smithi073 ceph-mon[66481]: failed to encode map e89 with expected crc 2024-03-11T22:32:29.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:29 smithi073 ceph-mon[66481]: osdmap e89: 8 total, 7 up, 8 in 2024-03-11T22:32:29.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:29 smithi073 ceph-mon[66481]: failed to encode map e89 with expected crc 2024-03-11T22:32:29.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:29 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:32:29.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:29 smithi099 ceph-mon[53203]: failed to encode map e89 with expected crc 2024-03-11T22:32:29.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:29 smithi099 ceph-mon[53203]: osdmap e89: 8 total, 7 up, 8 in 2024-03-11T22:32:29.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:29 smithi099 ceph-mon[53203]: failed to encode map e89 with expected crc 2024-03-11T22:32:30.438 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:30 smithi073 podman[74874]: 2024-03-11 22:32:30.217567726 +0000 UTC m=+0.033519847 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:32:30.438 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:30 smithi073 podman[74874]: 2024-03-11 22:32:30.397445927 +0000 UTC m=+0.213398038 container create dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: pgmap v96: 97 pgs: 26 peering, 4 stale+active+clean, 67 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.1 MiB/s wr, 465 op/s 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: osdmap e90: 8 total, 7 up, 8 in 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.702 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.703 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:30 smithi073 ceph-mon[66481]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.716 INFO:teuthology.orchestra.run.smithi073.stdout: 5 212822 10.72 MB/sec execute 346 sec latency 1020.657 ms 2024-03-11T22:32:30.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:30 smithi073 podman[74874]: 2024-03-11 22:32:30.768679061 +0000 UTC m=+0.584631183 container init dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:32:30.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:30 smithi073 podman[74874]: 2024-03-11 22:32:30.774828274 +0000 UTC m=+0.590780386 container start dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T22:32:30.953 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:30 smithi073 podman[74874]: 2024-03-11 22:32:30.94476647 +0000 UTC m=+0.760718578 container attach dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5) 2024-03-11T22:32:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: pgmap v96: 97 pgs: 26 peering, 4 stale+active+clean, 67 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.1 MiB/s wr, 465 op/s 2024-03-11T22:32:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:32:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: osdmap e90: 8 total, 7 up, 8 in 2024-03-11T22:32:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:30 smithi099 ceph-mon[53203]: failed to encode map e90 with expected crc 2024-03-11T22:32:31.459 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:31 smithi073 podman[74874]: 2024-03-11 22:32:31.203629435 +0000 UTC m=+1.019581542 container died dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T22:32:31.717 INFO:teuthology.orchestra.run.smithi073.stdout: 5 213338 10.74 MB/sec execute 347 sec latency 37.665 ms 2024-03-11T22:32:32.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:32 smithi073 podman[74899]: 2024-03-11 22:32:32.422142236 +0000 UTC m=+1.204643184 container remove dd1e966a7154ffffc239c9f61f865c8b17270278d0fe09c8ff89e21547286c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:32:32.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:32 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-11T22:32:32.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:32 smithi073 systemd[1]: Stopped Ceph osd.1 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:32:32.513 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:32 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.1.service: Consumed 3min 40.955s CPU time. 2024-03-11T22:32:32.717 INFO:teuthology.orchestra.run.smithi073.stdout: 5 213724 10.75 MB/sec execute 348 sec latency 28.613 ms 2024-03-11T22:32:32.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:32 smithi073 ceph-mon[66481]: pgmap v98: 97 pgs: 39 peering, 58 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 500 op/s 2024-03-11T22:32:32.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:32 smithi099 ceph-mon[53203]: pgmap v98: 97 pgs: 39 peering, 58 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 500 op/s 2024-03-11T22:32:33.016 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:32 smithi073 systemd[1]: Starting Ceph osd.1 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:32:33.278 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:33 smithi073 podman[74998]: 2024-03-11 22:32:32.918070709 +0000 UTC m=+0.020661940 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:32:33.278 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:33 smithi073 podman[74998]: 2024-03-11 22:32:33.039155094 +0000 UTC m=+0.141746304 container create 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T22:32:33.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 214017 10.73 MB/sec execute 349 sec latency 55.821 ms 2024-03-11T22:32:33.797 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:33 smithi073 podman[74998]: 2024-03-11 22:32:33.63582521 +0000 UTC m=+0.738416428 container init 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T22:32:33.797 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:33 smithi073 podman[74998]: 2024-03-11 22:32:33.640310958 +0000 UTC m=+0.742902180 container start 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:32:33.797 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:33 smithi073 podman[74998]: 2024-03-11 22:32:33.780841073 +0000 UTC m=+0.883432293 container attach 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T22:32:34.311 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 bash[74998]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate[75012]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-11T22:32:34.312 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:34 smithi073 podman[74998]: 2024-03-11 22:32:34.286375177 +0000 UTC m=+1.388966396 container died 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T22:32:34.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 214270 10.71 MB/sec execute 350 sec latency 47.942 ms 2024-03-11T22:32:34.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:34 smithi073 ceph-mon[66481]: pgmap v99: 97 pgs: 39 peering, 58 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 290 op/s 2024-03-11T22:32:34.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:34 smithi099 ceph-mon[53203]: pgmap v99: 97 pgs: 39 peering, 58 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 290 op/s 2024-03-11T22:32:35.370 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:35 smithi073 podman[75149]: 2024-03-11 22:32:35.186684174 +0000 UTC m=+0.894225156 container remove 983bfb0e7867c4e57eb69c527933c9608551fa1191f967d14fc91bf39365452d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T22:32:35.625 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:35 smithi073 podman[75208]: 2024-03-11 22:32:35.339052069 +0000 UTC m=+0.023261583 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:32:35.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 214497 10.70 MB/sec execute 351 sec latency 54.858 ms 2024-03-11T22:32:35.876 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:35 smithi073 podman[75208]: 2024-03-11 22:32:35.755519944 +0000 UTC m=+0.439729446 container create 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:32:35.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:35 smithi073 ceph-mon[66481]: pgmap v100: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.2 MiB/s wr, 621 op/s; 192/1722 objects degraded (11.150%) 2024-03-11T22:32:35.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:35 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 192/1722 objects degraded (11.150%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:35.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:35 smithi073 ceph-mon[66481]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-11T22:32:36.129 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 podman[75208]: 2024-03-11 22:32:36.118241289 +0000 UTC m=+0.802450810 container init 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T22:32:36.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:35 smithi099 ceph-mon[53203]: pgmap v100: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.2 MiB/s wr, 621 op/s; 192/1722 objects degraded (11.150%) 2024-03-11T22:32:36.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:35 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 192/1722 objects degraded (11.150%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:36.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:35 smithi099 ceph-mon[53203]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-11T22:32:36.380 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 podman[75208]: 2024-03-11 22:32:36.125321305 +0000 UTC m=+0.809530818 container start 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:32:36.380 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 ceph-osd[75226]: -- 172.21.15.73:0/3191952466 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ddd7b7f1e0 con 0x55ddd7b51000 2024-03-11T22:32:36.380 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 bash[75208]: 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 2024-03-11T22:32:36.380 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 systemd[1]: Started Ceph osd.1 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:32:36.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 214903 10.70 MB/sec execute 352 sec latency 57.208 ms 2024-03-11T22:32:36.898 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:36 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:32:36.701+0000 7f96541637c0 -1 Falling back to public interface 2024-03-11T22:32:37.426 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:37.426 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:37.426 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:37.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 215373 10.68 MB/sec execute 353 sec latency 49.231 ms 2024-03-11T22:32:37.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:37.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:37.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:38.436 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:38 smithi073 ceph-mon[66481]: pgmap v101: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 556 op/s; 192/1722 objects degraded (11.150%) 2024-03-11T22:32:38.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 215824 10.68 MB/sec execute 354 sec latency 48.029 ms 2024-03-11T22:32:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:38 smithi099 ceph-mon[53203]: pgmap v101: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.6 MiB/s wr, 556 op/s; 192/1722 objects degraded (11.150%) 2024-03-11T22:32:39.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 216399 10.71 MB/sec execute 355 sec latency 36.722 ms 2024-03-11T22:32:39.731 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:39.731 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:39.731 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:39 smithi073 ceph-mon[66481]: pgmap v102: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 608 op/s; 182/1665 objects degraded (10.931%) 2024-03-11T22:32:39.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:39.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:39.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:39 smithi099 ceph-mon[53203]: pgmap v102: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 608 op/s; 182/1665 objects degraded (10.931%) 2024-03-11T22:32:40.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 216919 10.74 MB/sec execute 356 sec latency 30.298 ms 2024-03-11T22:32:41.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 217278 10.74 MB/sec execute 357 sec latency 47.166 ms 2024-03-11T22:32:41.753 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:41 smithi073 ceph-mon[66481]: pgmap v103: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 448 op/s; 179/1650 objects degraded (10.848%) 2024-03-11T22:32:41.753 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:41 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 179/1650 objects degraded (10.848%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:41 smithi099 ceph-mon[53203]: pgmap v103: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 448 op/s; 179/1650 objects degraded (10.848%) 2024-03-11T22:32:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:41 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 179/1650 objects degraded (10.848%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:42.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 217575 10.73 MB/sec execute 358 sec latency 44.512 ms 2024-03-11T22:32:43.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 217848 10.71 MB/sec execute 359 sec latency 43.648 ms 2024-03-11T22:32:44.290 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:44 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:32:44.182+0000 7f96541637c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-11T22:32:44.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 218095 10.69 MB/sec execute 360 sec latency 47.308 ms 2024-03-11T22:32:44.800 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:44 smithi073 ceph-mon[66481]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 179/1650 objects degraded (10.848%) 2024-03-11T22:32:44.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:44 smithi099 ceph-mon[53203]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 340 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 179/1650 objects degraded (10.848%) 2024-03-11T22:32:45.555 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:45 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:32:45.417+0000 7f96541637c0 -1 osd.1 88 log_to_monitors true 2024-03-11T22:32:45.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 218444 10.69 MB/sec execute 361 sec latency 51.022 ms 2024-03-11T22:32:45.829 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:32:45 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:32:45.736+0000 7f96455bb700 -1 osd.1 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:32:45.829 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:45 smithi073 ceph-mon[66481]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:32:45.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:45 smithi099 ceph-mon[53203]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:32:46.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 218837 10.68 MB/sec execute 362 sec latency 49.741 ms 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: pgmap v105: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.5 MiB/s wr, 698 op/s; 199/1740 objects degraded (11.437%) 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: failed to encode map e91 with expected crc 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: osdmap e91: 8 total, 7 up, 8 in 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:32:46.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: failed to encode map e91 with expected crc 2024-03-11T22:32:46.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:46.847 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:46.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: pgmap v105: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 348 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.5 MiB/s wr, 698 op/s; 199/1740 objects degraded (11.437%) 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: failed to encode map e91 with expected crc 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: osdmap e91: 8 total, 7 up, 8 in 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: from='osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: failed to encode map e91 with expected crc 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:47.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 218900 10.65 MB/sec execute 363 sec latency 969.658 ms 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: failed to encode map e92 with expected crc 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483] boot 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: failed to encode map e92 with expected crc 2024-03-11T22:32:47.945 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:47.946 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:47 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:47.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:32:47.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: failed to encode map e92 with expected crc 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: osd.1 [v2:172.21.15.73:6810/4277680483,v1:172.21.15.73:6811/4277680483] boot 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: failed to encode map e92 with expected crc 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:47 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:48.718 INFO:teuthology.orchestra.run.smithi073.stdout: 5 218975 10.62 MB/sec execute 364 sec latency 1969.699 ms 2024-03-11T22:32:48.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: pgmap v108: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.1 MiB/s wr, 475 op/s; 199/1740 objects degraded (11.437%) 2024-03-11T22:32:48.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T22:32:48.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: pgmap v108: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.1 MiB/s wr, 475 op/s; 199/1740 objects degraded (11.437%) 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: failed to encode map e93 with expected crc 2024-03-11T22:32:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:48 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 199/1740 objects degraded (11.437%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: failed to encode map e93 with expected crc 2024-03-11T22:32:49.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:48 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 199/1740 objects degraded (11.437%), 24 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:49.718 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:49 smithi073 ceph-mon[66481]: pgmap v111: 97 pgs: 1 activating+undersized+degraded+remapped, 6 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+undersized+degraded, 73 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 722 KiB/s wr, 88 op/s; 168/1530 objects degraded (10.980%); 3.1 KiB/s, 9 objects/s recovering 2024-03-11T22:32:49.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 219023 10.60 MB/sec execute 365 sec latency 2791.148 ms 2024-03-11T22:32:49.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: failed to encode map e94 with expected crc 2024-03-11T22:32:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:49 smithi099 ceph-mon[53203]: pgmap v111: 97 pgs: 1 activating+undersized+degraded+remapped, 6 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+undersized+degraded, 73 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 722 KiB/s wr, 88 op/s; 168/1530 objects degraded (10.980%); 3.1 KiB/s, 9 objects/s recovering 2024-03-11T22:32:50.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 219469 10.60 MB/sec execute 366 sec latency 31.250 ms 2024-03-11T22:32:50.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:50 smithi073 ceph-mon[66481]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:50 smithi073 ceph-mon[66481]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T22:32:50.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:50 smithi073 ceph-mon[66481]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:50 smithi073 ceph-mon[66481]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:50 smithi073 ceph-mon[66481]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:50 smithi099 ceph-mon[53203]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:50 smithi099 ceph-mon[53203]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T22:32:50.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:50 smithi099 ceph-mon[53203]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:50 smithi099 ceph-mon[53203]: failed to encode map e95 with expected crc 2024-03-11T22:32:50.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:50 smithi099 ceph-mon[53203]: failed to encode map e95 with expected crc 2024-03-11T22:32:51.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 219845 10.61 MB/sec execute 367 sec latency 890.896 ms 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.932 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: pgmap v114: 97 pgs: 1 peering, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 1.8 MiB/s wr, 352 op/s; 167/1416 objects degraded (11.794%); 3.1 KiB/s, 60 keys/s, 16 objects/s recovering 2024-03-11T22:32:51.933 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:51 smithi073 ceph-mon[66481]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T22:32:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T22:32:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: failed to encode map e96 with expected crc 2024-03-11T22:32:51.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: pgmap v114: 97 pgs: 1 peering, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 1.8 MiB/s wr, 352 op/s; 167/1416 objects degraded (11.794%); 3.1 KiB/s, 60 keys/s, 16 objects/s recovering 2024-03-11T22:32:51.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:51 smithi099 ceph-mon[53203]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T22:32:52.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 220260 10.62 MB/sec execute 368 sec latency 960.962 ms 2024-03-11T22:32:52.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.957 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:32:52.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:32:52.959 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:52 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-11T22:32:52.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: failed to encode map e97 with expected crc 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:52.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:32:52.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:32:52.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:52 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-11T22:32:53.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 220727 10.64 MB/sec execute 369 sec latency 994.680 ms 2024-03-11T22:32:53.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T22:32:53.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 167/1416 objects degraded (11.794%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: pgmap v117: 97 pgs: 1 peering, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 263 op/s; 167/1416 objects degraded (11.794%); 22 B/s, 60 keys/s, 7 objects/s recovering 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:53.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:53.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T22:32:53.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: failed to encode map e98 with expected crc 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 167/1416 objects degraded (11.794%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: pgmap v117: 97 pgs: 1 peering, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.1 MiB/s wr, 263 op/s; 167/1416 objects degraded (11.794%); 22 B/s, 60 keys/s, 7 objects/s recovering 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:32:53.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:54.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 221064 10.64 MB/sec execute 370 sec latency 44.922 ms 2024-03-11T22:32:54.960 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:32:55.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 221471 10.63 MB/sec execute 371 sec latency 51.948 ms 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (5m) 10s ago 14m 15.7M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (107s) 10s ago 14m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (100s) 95s ago 14m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (5m) 10s ago 14m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (8m) 10s ago 8m 38.1M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (8m) 95s ago 8m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:32:55.954 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (4m) 10s ago 15m 500M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (3m) 95s ago 13m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (3m) 10s ago 16m 47.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (2m) 95s ago 13m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (8m) 10s ago 8m 276M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (8m) 95s ago 8m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (5m) 10s ago 14m 13.5M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (4m) 95s ago 13m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (66s) 10s ago 13m 236M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (20s) 10s ago 12m 12.7M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (12m) 10s ago 12m 535M 1936M 17.2.0 e1d6a67b021e 7202d161bb20 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (11m) 10s ago 11m 406M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (11m) 95s ago 11m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (10m) 95s ago 10m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (10m) 95s ago 10m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (10m) 95s ago 10m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:32:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (3m) 10s ago 13m 60.8M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:32:56.581 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:32:56.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:32:56.583 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:32:56.583 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-11T22:32:56.583 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 6 2024-03-11T22:32:56.583 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:32:56.583 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:32:56.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 221750 10.61 MB/sec execute 372 sec latency 47.979 ms 2024-03-11T22:32:56.757 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:56 smithi073 ceph-mon[66481]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:56.757 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:56 smithi073 ceph-mon[66481]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:56.757 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:56 smithi073 ceph-mon[66481]: pgmap v118: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.6 MiB/s wr, 476 op/s; 54/1707 objects degraded (3.163%); 691 KiB/s, 40 keys/s, 21 objects/s recovering 2024-03-11T22:32:56.757 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:56 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2069449834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:56.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:56 smithi099 ceph-mon[53203]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:56 smithi099 ceph-mon[53203]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:56 smithi099 ceph-mon[53203]: pgmap v118: 97 pgs: 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.6 MiB/s wr, 476 op/s; 54/1707 objects degraded (3.163%); 691 KiB/s, 40 keys/s, 21 objects/s recovering 2024-03-11T22:32:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:56 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2069449834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:32:57.083 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "8/23 daemons upgraded", 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:32:57.084 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:32:57.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 222046 10.61 MB/sec execute 373 sec latency 39.088 ms 2024-03-11T22:32:57.728 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:32:57.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:57 smithi073 ceph-mon[66481]: from='client.44195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:57.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:57 smithi099 ceph-mon[53203]: from='client.44195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:58.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 222465 10.61 MB/sec execute 374 sec latency 53.755 ms 2024-03-11T22:32:58.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:58 smithi073 ceph-mon[66481]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:58.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:58 smithi073 ceph-mon[66481]: pgmap v119: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 5.2 MiB/s wr, 421 op/s; 1.2 MiB/s, 34 keys/s, 26 objects/s recovering 2024-03-11T22:32:58.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:58 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/1707 objects degraded (3.163%), 2 pgs degraded) 2024-03-11T22:32:58.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:58 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:32:58.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:58 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2468502963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:32:58.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:58 smithi099 ceph-mon[53203]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:32:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:58 smithi099 ceph-mon[53203]: pgmap v119: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 5.2 MiB/s wr, 421 op/s; 1.2 MiB/s, 34 keys/s, 26 objects/s recovering 2024-03-11T22:32:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:58 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/1707 objects degraded (3.163%), 2 pgs degraded) 2024-03-11T22:32:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:58 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:32:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:58 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2468502963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:32:59.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 222998 10.61 MB/sec execute 375 sec latency 38.718 ms 2024-03-11T22:32:59.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:32:59 smithi073 ceph-mon[66481]: pgmap v120: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.9 MiB/s wr, 659 op/s; 1024 KiB/s, 21 objects/s recovering 2024-03-11T22:32:59.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:32:59 smithi099 ceph-mon[53203]: pgmap v120: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.9 MiB/s wr, 659 op/s; 1024 KiB/s, 21 objects/s recovering 2024-03-11T22:33:00.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 223343 10.61 MB/sec execute 376 sec latency 44.281 ms 2024-03-11T22:33:01.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 223805 10.62 MB/sec execute 377 sec latency 97.862 ms 2024-03-11T22:33:02.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 224096 10.61 MB/sec execute 378 sec latency 49.759 ms 2024-03-11T22:33:02.931 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:02 smithi073 ceph-mon[66481]: pgmap v121: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.0 MiB/s wr, 572 op/s; 825 KiB/s, 17 objects/s recovering 2024-03-11T22:33:02.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:02 smithi099 ceph-mon[53203]: pgmap v121: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.0 MiB/s wr, 572 op/s; 825 KiB/s, 17 objects/s recovering 2024-03-11T22:33:03.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 224322 10.61 MB/sec execute 379 sec latency 60.012 ms 2024-03-11T22:33:04.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 224544 10.61 MB/sec execute 380 sec latency 41.758 ms 2024-03-11T22:33:04.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:04 smithi073 ceph-mon[66481]: pgmap v122: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.4 MiB/s wr, 520 op/s; 750 KiB/s, 15 objects/s recovering 2024-03-11T22:33:04.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:04 smithi099 ceph-mon[53203]: pgmap v122: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.4 MiB/s wr, 520 op/s; 750 KiB/s, 15 objects/s recovering 2024-03-11T22:33:04.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:05.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 224748 10.59 MB/sec execute 381 sec latency 69.251 ms 2024-03-11T22:33:06.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 225230 10.58 MB/sec execute 382 sec latency 37.832 ms 2024-03-11T22:33:06.938 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:06 smithi073 ceph-mon[66481]: pgmap v123: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.2 MiB/s wr, 668 op/s; 683 KiB/s, 14 objects/s recovering 2024-03-11T22:33:06.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:06 smithi099 ceph-mon[53203]: pgmap v123: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.2 MiB/s wr, 668 op/s; 683 KiB/s, 14 objects/s recovering 2024-03-11T22:33:07.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 225613 10.58 MB/sec execute 383 sec latency 37.716 ms 2024-03-11T22:33:07.951 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:07 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:33:07.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:07 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:33:08.706 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:08 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:33:08.706 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:08 smithi073 ceph-mon[66481]: Upgrade: osd.2 is safe to restart 2024-03-11T22:33:08.706 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:08 smithi073 ceph-mon[66481]: pgmap v124: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 439 op/s; 341 KiB/s, 4 objects/s recovering 2024-03-11T22:33:08.719 INFO:teuthology.orchestra.run.smithi073.stdout: 5 226147 10.59 MB/sec execute 384 sec latency 55.790 ms 2024-03-11T22:33:08.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:08 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-11T22:33:08.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:08 smithi099 ceph-mon[53203]: Upgrade: osd.2 is safe to restart 2024-03-11T22:33:08.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:08 smithi099 ceph-mon[53203]: pgmap v124: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 439 op/s; 341 KiB/s, 4 objects/s recovering 2024-03-11T22:33:09.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 226786 10.60 MB/sec execute 385 sec latency 33.638 ms 2024-03-11T22:33:09.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:09.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:09.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:09 smithi073 ceph-mon[66481]: pgmap v125: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-11T22:33:09.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:09.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:09.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:09 smithi099 ceph-mon[53203]: pgmap v125: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 551 op/s 2024-03-11T22:33:10.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 227358 10.62 MB/sec execute 386 sec latency 52.675 ms 2024-03-11T22:33:11.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:11 smithi073 ceph-mon[66481]: Upgrade: Updating osd.2 2024-03-11T22:33:11.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:11.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:33:11.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:11.228 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:11 smithi073 ceph-mon[66481]: Deploying daemon osd.2 on smithi073 2024-03-11T22:33:11.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:11 smithi099 ceph-mon[53203]: Upgrade: Updating osd.2 2024-03-11T22:33:11.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:11.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:33:11.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:11.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:11 smithi099 ceph-mon[53203]: Deploying daemon osd.2 on smithi073 2024-03-11T22:33:11.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 227726 10.62 MB/sec execute 387 sec latency 58.198 ms 2024-03-11T22:33:12.236 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:12 smithi073 ceph-mon[66481]: pgmap v126: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-03-11T22:33:12.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:12 smithi099 ceph-mon[53203]: pgmap v126: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 519 op/s 2024-03-11T22:33:12.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 228100 10.64 MB/sec execute 388 sec latency 42.265 ms 2024-03-11T22:33:13.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 228381 10.62 MB/sec execute 389 sec latency 53.659 ms 2024-03-11T22:33:14.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 228664 10.60 MB/sec execute 390 sec latency 65.417 ms 2024-03-11T22:33:14.835 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:14 smithi073 ceph-mon[66481]: pgmap v127: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-11T22:33:14.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:14 smithi099 ceph-mon[53203]: pgmap v127: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 485 op/s 2024-03-11T22:33:15.604 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:15 smithi073 systemd[1]: Stopping Ceph osd.2 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:33:15.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 228824 10.58 MB/sec execute 391 sec latency 55.075 ms 2024-03-11T22:33:16.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:15 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[42982]: 2024-03-11T22:33:15.862+0000 7fec0cc26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:16.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:15 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[42982]: 2024-03-11T22:33:15.862+0000 7fec0cc26700 -1 osd.2 98 *** Got signal Terminated *** 2024-03-11T22:33:16.106 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:15 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[42982]: 2024-03-11T22:33:15.862+0000 7fec0cc26700 -1 osd.2 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:33:16.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 228996 10.57 MB/sec execute 392 sec latency 66.062 ms 2024-03-11T22:33:16.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:16 smithi073 ceph-mon[66481]: pgmap v128: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-03-11T22:33:16.876 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:16 smithi073 ceph-mon[66481]: osd.2 marked itself down and dead 2024-03-11T22:33:16.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:16 smithi099 ceph-mon[53203]: pgmap v128: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-03-11T22:33:16.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:16 smithi099 ceph-mon[53203]: osd.2 marked itself down and dead 2024-03-11T22:33:17.469 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:17 smithi073 podman[77340]: 2024-03-11 22:33:17.356482323 +0000 UTC m=+1.634723424 container died 7202d161bb206aac41452757bab21cb2e2f95baefefee32b8db61471409ac4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, distribution-scope=public, version=8, GIT_BRANCH=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-03-11T22:33:17.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 229189 10.56 MB/sec execute 393 sec latency 1004.887 ms 2024-03-11T22:33:17.931 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:17 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:33:17.931 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:17 smithi073 ceph-mon[66481]: failed to encode map e99 with expected crc 2024-03-11T22:33:17.931 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:17 smithi073 ceph-mon[66481]: osdmap e99: 8 total, 7 up, 8 in 2024-03-11T22:33:17.931 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:17 smithi073 ceph-mon[66481]: failed to encode map e99 with expected crc 2024-03-11T22:33:17.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:17 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:33:17.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:17 smithi099 ceph-mon[53203]: failed to encode map e99 with expected crc 2024-03-11T22:33:17.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:17 smithi099 ceph-mon[53203]: osdmap e99: 8 total, 7 up, 8 in 2024-03-11T22:33:17.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:17 smithi099 ceph-mon[53203]: failed to encode map e99 with expected crc 2024-03-11T22:33:18.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:18 smithi073 podman[77340]: 2024-03-11 22:33:18.077464751 +0000 UTC m=+2.355705866 container cleanup 7202d161bb206aac41452757bab21cb2e2f95baefefee32b8db61471409ac4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vendor=Red Hat, Inc., version=8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, io.k8s.display-name=CentOS Stream 8) 2024-03-11T22:33:18.186 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:18 smithi073 bash[77340]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:33:18.690 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:18 smithi073 podman[77358]: 2024-03-11 22:33:18.442406636 +0000 UTC m=+1.086534200 container remove 7202d161bb206aac41452757bab21cb2e2f95baefefee32b8db61471409ac4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , ceph=True, release=754, architecture=x86_64, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8) 2024-03-11T22:33:18.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 229516 10.56 MB/sec execute 394 sec latency 1019.957 ms 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: pgmap v130: 97 pgs: 8 stale+active+clean, 89 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: osdmap e100: 8 total, 7 up, 8 in 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.948 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:18 smithi073 ceph-mon[66481]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: pgmap v130: 97 pgs: 8 stale+active+clean, 89 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: osdmap e100: 8 total, 7 up, 8 in 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:18.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:18 smithi099 ceph-mon[53203]: failed to encode map e100 with expected crc 2024-03-11T22:33:19.145 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:18 smithi073 podman[77447]: 2024-03-11 22:33:18.894348878 +0000 UTC m=+0.025968414 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:19.145 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:19 smithi073 podman[77447]: 2024-03-11 22:33:19.051074472 +0000 UTC m=+0.182693984 container create 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T22:33:19.467 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:19 smithi073 podman[77447]: 2024-03-11 22:33:19.414255583 +0000 UTC m=+0.545875112 container init 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:19.467 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:19 smithi073 podman[77447]: 2024-03-11 22:33:19.425352218 +0000 UTC m=+0.556971731 container start 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-11T22:33:19.719 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:19 smithi073 podman[77447]: 2024-03-11 22:33:19.606799753 +0000 UTC m=+0.738419274 container attach 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T22:33:19.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 229831 10.55 MB/sec execute 395 sec latency 60.827 ms 2024-03-11T22:33:19.971 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:19 smithi073 podman[77447]: 2024-03-11 22:33:19.867959498 +0000 UTC m=+0.999579022 container died 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T22:33:19.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:19 smithi073 ceph-mon[66481]: pgmap v132: 97 pgs: 2 active+undersized, 8 active+undersized+degraded, 7 stale+active+clean, 80 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 370 op/s; 62/1611 objects degraded (3.849%) 2024-03-11T22:33:19.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:19 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 62/1611 objects degraded (3.849%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:20.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:19 smithi099 ceph-mon[53203]: pgmap v132: 97 pgs: 2 active+undersized, 8 active+undersized+degraded, 7 stale+active+clean, 80 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 370 op/s; 62/1611 objects degraded (3.849%) 2024-03-11T22:33:20.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:19 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 62/1611 objects degraded (3.849%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:20.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 230236 10.54 MB/sec execute 396 sec latency 34.503 ms 2024-03-11T22:33:21.010 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:20 smithi073 podman[77473]: 2024-03-11 22:33:20.974255919 +0000 UTC m=+1.097383301 container remove 35e1e36299cd2d0ee463501f9566258a9bfbd067862b2b5c5ba97af4b0761a54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-11T22:33:21.265 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-11T22:33:21.265 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 systemd[1]: Stopped Ceph osd.2 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:33:21.265 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.2.service: Consumed 2min 55.048s CPU time. 2024-03-11T22:33:21.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 systemd[1]: Starting Ceph osd.2 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:33:21.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 230480 10.52 MB/sec execute 397 sec latency 72.062 ms 2024-03-11T22:33:21.768 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 podman[77572]: 2024-03-11 22:33:21.478367879 +0000 UTC m=+0.022847702 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:21.768 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 podman[77572]: 2024-03-11 22:33:21.643003704 +0000 UTC m=+0.187483511 container create a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T22:33:22.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 podman[77572]: 2024-03-11 22:33:21.98062591 +0000 UTC m=+0.525105733 container init a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T22:33:22.021 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:21 smithi073 podman[77572]: 2024-03-11 22:33:21.98926881 +0000 UTC m=+0.533748618 container start a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T22:33:22.282 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 podman[77572]: 2024-03-11 22:33:22.126698205 +0000 UTC m=+0.671178020 container attach a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T22:33:22.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 230829 10.53 MB/sec execute 398 sec latency 46.310 ms 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate[77587]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-11T22:33:22.795 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-11T22:33:22.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 bash[77572]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-11T22:33:22.796 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:22 smithi073 podman[77572]: 2024-03-11 22:33:22.633308006 +0000 UTC m=+1.177787812 container died a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T22:33:22.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:22 smithi073 ceph-mon[66481]: pgmap v133: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 569 op/s; 143/1533 objects degraded (9.328%) 2024-03-11T22:33:22.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:22 smithi099 ceph-mon[53203]: pgmap v133: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 569 op/s; 143/1533 objects degraded (9.328%) 2024-03-11T22:33:23.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 231159 10.53 MB/sec execute 399 sec latency 45.491 ms 2024-03-11T22:33:23.807 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:23 smithi073 ceph-mon[66481]: pgmap v134: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.8 MiB/s wr, 373 op/s; 143/1533 objects degraded (9.328%) 2024-03-11T22:33:23.807 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:23.807 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:24.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:23 smithi099 ceph-mon[53203]: pgmap v134: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.8 MiB/s wr, 373 op/s; 143/1533 objects degraded (9.328%) 2024-03-11T22:33:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:24.609 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:24 smithi073 podman[77708]: 2024-03-11 22:33:24.441558967 +0000 UTC m=+1.794560540 container remove a3a33f4138758bd91f3583447a500bd73cd09ee7ff01c824355cf57a7f87045d (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T22:33:24.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 231528 10.55 MB/sec execute 400 sec latency 57.349 ms 2024-03-11T22:33:24.867 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:24 smithi073 podman[77763]: 2024-03-11 22:33:24.631826316 +0000 UTC m=+0.029179624 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:24.867 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:24 smithi073 podman[77763]: 2024-03-11 22:33:24.7512363 +0000 UTC m=+0.148589584 container create 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:33:25.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 podman[77763]: 2024-03-11 22:33:25.089740871 +0000 UTC m=+0.487094169 container init 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:33:25.119 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 podman[77763]: 2024-03-11 22:33:25.098105108 +0000 UTC m=+0.495458396 container start 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T22:33:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 ceph-osd[77780]: -- 172.21.15.73:0/981477292 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588581b91e0 con 0x55885818b000 2024-03-11T22:33:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 bash[77763]: 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f 2024-03-11T22:33:25.371 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 systemd[1]: Started Ceph osd.2 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:33:25.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 231797 10.53 MB/sec execute 401 sec latency 58.691 ms 2024-03-11T22:33:25.884 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:25 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:33:25.685+0000 7fb50bd947c0 -1 Falling back to public interface 2024-03-11T22:33:26.398 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:26.398 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:26.398 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:26.398 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:26 smithi073 ceph-mon[66481]: pgmap v135: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 471 op/s; 144/1617 objects degraded (8.905%) 2024-03-11T22:33:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 232112 10.52 MB/sec execute 402 sec latency 35.037 ms 2024-03-11T22:33:26.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:26.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:26.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:26.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:26 smithi099 ceph-mon[53203]: pgmap v135: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 471 op/s; 144/1617 objects degraded (8.905%) 2024-03-11T22:33:27.407 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:27 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 144/1617 objects degraded (8.905%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 232461 10.52 MB/sec execute 403 sec latency 40.826 ms 2024-03-11T22:33:27.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:27 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 144/1617 objects degraded (8.905%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:28.216 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:33:28.424 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:28 smithi073 ceph-mon[66481]: pgmap v136: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 436 op/s; 144/1626 objects degraded (8.856%) 2024-03-11T22:33:28.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 232708 10.51 MB/sec execute 404 sec latency 74.509 ms 2024-03-11T22:33:28.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:28 smithi099 ceph-mon[53203]: pgmap v136: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 436 op/s; 144/1626 objects degraded (8.856%) 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (6m) 43s ago 15m 15.7M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (2m) 43s ago 15m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (2m) 2m ago 14m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (5m) 43s ago 14m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (9m) 43s ago 9m 38.1M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (9m) 2m ago 9m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (4m) 43s ago 16m 500M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (4m) 2m ago 14m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:33:29.339 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (3m) 43s ago 16m 47.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (3m) 2m ago 14m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (8m) 43s ago 8m 276M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (8m) 2m ago 8m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (5m) 43s ago 14m 13.5M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (5m) 2m ago 14m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (100s) 43s ago 13m 236M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (53s) 43s ago 13m 12.7M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 starting - - - 1936M 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (12m) 43s ago 12m 406M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (11m) 2m ago 11m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (11m) 2m ago 11m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (11m) 2m ago 11m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (10m) 2m ago 10m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:33:29.340 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (4m) 43s ago 13m 60.8M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:33:29.468 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:29 smithi073 ceph-mon[66481]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:29.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 232982 10.50 MB/sec execute 405 sec latency 84.422 ms 2024-03-11T22:33:29.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:29 smithi099 ceph-mon[53203]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:33:30.019 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 6 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:33:30.020 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:33:30.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:30 smithi073 ceph-mon[66481]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:30.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:30 smithi073 ceph-mon[66481]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:30.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:30 smithi073 ceph-mon[66481]: pgmap v137: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.5 MiB/s wr, 399 op/s; 150/1680 objects degraded (8.929%) 2024-03-11T22:33:30.477 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:30 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1504910200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "8/23 daemons upgraded", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:33:30.545 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:33:30.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 233256 10.49 MB/sec execute 406 sec latency 66.705 ms 2024-03-11T22:33:30.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:30 smithi099 ceph-mon[53203]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:30.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:30 smithi099 ceph-mon[53203]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:30.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:30 smithi099 ceph-mon[53203]: pgmap v137: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.5 MiB/s wr, 399 op/s; 150/1680 objects degraded (8.929%) 2024-03-11T22:33:30.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:30 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1504910200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 150/1680 objects degraded (8.929%), 19 pgs degraded 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: osd.2 (root=default,host=smithi073) is down 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 150/1680 objects degraded (8.929%), 19 pgs degraded 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-03-11T22:33:31.260 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-03-11T22:33:31.261 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-03-11T22:33:31.485 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:31 smithi073 ceph-mon[66481]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:31.485 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:31 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2957145672' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:33:31.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 233572 10.47 MB/sec execute 407 sec latency 72.353 ms 2024-03-11T22:33:31.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:31 smithi099 ceph-mon[53203]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:33:31.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:31 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2957145672' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:33:32.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:32 smithi073 ceph-mon[66481]: pgmap v138: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 547 op/s; 153/1695 objects degraded (9.027%) 2024-03-11T22:33:32.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:32 smithi099 ceph-mon[53203]: pgmap v138: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 547 op/s; 153/1695 objects degraded (9.027%) 2024-03-11T22:33:32.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 233867 10.47 MB/sec execute 408 sec latency 40.877 ms 2024-03-11T22:33:32.758 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:32 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:33:32.589+0000 7fb50bd947c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-11T22:33:33.515 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:33 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 153/1695 objects degraded (9.027%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:33.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234240 10.47 MB/sec execute 409 sec latency 72.132 ms 2024-03-11T22:33:33.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:33 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 153/1695 objects degraded (9.027%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:33.773 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:33 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:33:33.610+0000 7fb50bd947c0 -1 osd.2 98 log_to_monitors true 2024-03-11T22:33:34.538 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:34 smithi073 ceph-mon[66481]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:33:34.538 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:34 smithi073 ceph-mon[66481]: pgmap v139: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 414 op/s; 153/1695 objects degraded (9.027%) 2024-03-11T22:33:34.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234526 10.47 MB/sec execute 410 sec latency 81.556 ms 2024-03-11T22:33:34.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:34 smithi099 ceph-mon[53203]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:33:34.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:34 smithi099 ceph-mon[53203]: pgmap v139: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 414 op/s; 153/1695 objects degraded (9.027%) 2024-03-11T22:33:35.046 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:33:34 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:33:34.858+0000 7fb4f91e4700 -1 osd.2 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:33:35.558 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:33:35.558 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.558 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: osdmap e101: 8 total, 7 up, 8 in 2024-03-11T22:33:35.558 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:33:35.558 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.559 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.559 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' 2024-03-11T22:33:35.559 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:35.559 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:35 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:35.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234855 10.47 MB/sec execute 411 sec latency 63.210 ms 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: osdmap e101: 8 total, 7 up, 8 in 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: failed to encode map e101 with expected crc 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: from='osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590]' entity='osd.2' 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:35 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:36.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:36 smithi073 ceph-mon[66481]: pgmap v141: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 412 op/s; 149/1659 objects degraded (8.981%) 2024-03-11T22:33:36.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:36 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:33:36.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:36 smithi073 ceph-mon[66481]: osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590] boot 2024-03-11T22:33:36.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:36 smithi073 ceph-mon[66481]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T22:33:36.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:36 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:33:36.720 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234925 10.45 MB/sec execute 412 sec latency 881.441 ms 2024-03-11T22:33:36.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:36 smithi099 ceph-mon[53203]: pgmap v141: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 412 op/s; 149/1659 objects degraded (8.981%) 2024-03-11T22:33:36.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:36 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:33:36.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:36 smithi099 ceph-mon[53203]: osd.2 [v2:172.21.15.73:6818/1107334590,v1:172.21.15.73:6819/1107334590] boot 2024-03-11T22:33:36.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:36 smithi099 ceph-mon[53203]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T22:33:36.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:36 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: failed to encode map e102 with expected crc 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:37.545 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:37 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:37.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234925 10.43 MB/sec execute 413 sec latency 1881.471 ms 2024-03-11T22:33:37.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: failed to encode map e102 with expected crc 2024-03-11T22:33:37.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:37.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:37.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T22:33:37.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:37.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:37 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.550 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 325 op/s; 150/1662 objects degraded (9.025%) 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:38.551 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:38 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 149/1659 objects degraded (8.981%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:38.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 234945 10.40 MB/sec execute 414 sec latency 2811.305 ms 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e103 with expected crc 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 325 op/s; 150/1662 objects degraded (9.025%) 2024-03-11T22:33:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:38.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T22:33:38.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:38.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:38 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 149/1659 objects degraded (8.981%), 19 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:39.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.577 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:39.578 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:39.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 235282 10.41 MB/sec execute 415 sec latency 40.039 ms 2024-03-11T22:33:39.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: failed to encode map e104 with expected crc 2024-03-11T22:33:39.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:39.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:40.606 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T22:33:40.606 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.606 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.607 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:40 smithi073 ceph-mon[66481]: pgmap v147: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 2 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 84 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 267 op/s; 86/1635 objects degraded (5.260%); 57 KiB/s, 12 objects/s recovering 2024-03-11T22:33:40.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 235574 10.39 MB/sec execute 416 sec latency 447.098 ms 2024-03-11T22:33:40.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: failed to encode map e105 with expected crc 2024-03-11T22:33:40.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:40 smithi099 ceph-mon[53203]: pgmap v147: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 2 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 84 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 267 op/s; 86/1635 objects degraded (5.260%); 57 KiB/s, 12 objects/s recovering 2024-03-11T22:33:41.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.505 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:41 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 235866 10.38 MB/sec execute 417 sec latency 998.227 ms 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: failed to encode map e106 with expected crc 2024-03-11T22:33:41.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:41.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:33:41.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:41.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:41 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:42.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T22:33:42.516 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-11T22:33:42.517 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:42 smithi073 ceph-mon[66481]: pgmap v150: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 3.4 MiB/s wr, 276 op/s; 27/1635 objects degraded (1.651%); 1.3 MiB/s, 34 objects/s recovering 2024-03-11T22:33:42.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 236202 10.38 MB/sec execute 418 sec latency 52.592 ms 2024-03-11T22:33:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: failed to encode map e107 with expected crc 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-11T22:33:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:42 smithi099 ceph-mon[53203]: pgmap v150: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 3.4 MiB/s wr, 276 op/s; 27/1635 objects degraded (1.651%); 1.3 MiB/s, 34 objects/s recovering 2024-03-11T22:33:43.587 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:43 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 27/1635 objects degraded (1.651%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:43.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 236613 10.38 MB/sec execute 419 sec latency 75.102 ms 2024-03-11T22:33:43.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:43 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 27/1635 objects degraded (1.651%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T22:33:44.613 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:44 smithi073 ceph-mon[66481]: pgmap v151: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 353 B/s rd, 2.4 MiB/s wr, 190 op/s; 27/1635 objects degraded (1.651%); 940 KiB/s, 23 objects/s recovering 2024-03-11T22:33:44.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 237081 10.38 MB/sec execute 420 sec latency 44.957 ms 2024-03-11T22:33:44.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:44 smithi099 ceph-mon[53203]: pgmap v151: 97 pgs: 1 active+recovery_wait+degraded, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 353 B/s rd, 2.4 MiB/s wr, 190 op/s; 27/1635 objects degraded (1.651%); 940 KiB/s, 23 objects/s recovering 2024-03-11T22:33:45.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 237433 10.37 MB/sec execute 421 sec latency 28.997 ms 2024-03-11T22:33:45.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:45 smithi073 ceph-mon[66481]: pgmap v152: 97 pgs: 1 peering, 96 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.1 MiB/s wr, 561 op/s; 836 KiB/s, 23 objects/s recovering 2024-03-11T22:33:45.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:45 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1635 objects degraded (1.651%), 3 pgs degraded) 2024-03-11T22:33:45.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:45 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:33:46.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:45 smithi099 ceph-mon[53203]: pgmap v152: 97 pgs: 1 peering, 96 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.1 MiB/s wr, 561 op/s; 836 KiB/s, 23 objects/s recovering 2024-03-11T22:33:46.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:45 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1635 objects degraded (1.651%), 3 pgs degraded) 2024-03-11T22:33:46.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:45 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:33:46.721 INFO:teuthology.orchestra.run.smithi073.stdout: 5 237937 10.39 MB/sec execute 422 sec latency 40.583 ms 2024-03-11T22:33:47.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 238365 10.40 MB/sec execute 423 sec latency 36.539 ms 2024-03-11T22:33:47.933 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:47 smithi073 ceph-mon[66481]: pgmap v153: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.4 MiB/s wr, 397 op/s; 1.2 MiB/s, 14 objects/s recovering 2024-03-11T22:33:48.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:47 smithi099 ceph-mon[53203]: pgmap v153: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.4 MiB/s wr, 397 op/s; 1.2 MiB/s, 14 objects/s recovering 2024-03-11T22:33:48.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 238733 10.41 MB/sec execute 424 sec latency 32.833 ms 2024-03-11T22:33:49.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 239061 10.39 MB/sec execute 425 sec latency 44.258 ms 2024-03-11T22:33:49.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:49 smithi073 ceph-mon[66481]: pgmap v154: 97 pgs: 97 active+clean; 350 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.7 MiB/s wr, 490 op/s; 576 KiB/s, 8 objects/s recovering 2024-03-11T22:33:50.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:49 smithi099 ceph-mon[53203]: pgmap v154: 97 pgs: 97 active+clean; 350 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.7 MiB/s wr, 490 op/s; 576 KiB/s, 8 objects/s recovering 2024-03-11T22:33:50.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 239399 10.39 MB/sec execute 426 sec latency 38.861 ms 2024-03-11T22:33:51.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 239590 10.37 MB/sec execute 427 sec latency 62.417 ms 2024-03-11T22:33:52.009 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:51 smithi073 ceph-mon[66481]: pgmap v155: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.5 MiB/s wr, 569 op/s; 410 KiB/s, 2 objects/s recovering 2024-03-11T22:33:52.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:51 smithi099 ceph-mon[53203]: pgmap v155: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.5 MiB/s wr, 569 op/s; 410 KiB/s, 2 objects/s recovering 2024-03-11T22:33:52.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 239861 10.37 MB/sec execute 428 sec latency 60.195 ms 2024-03-11T22:33:53.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 240223 10.37 MB/sec execute 429 sec latency 53.717 ms 2024-03-11T22:33:54.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 240622 10.36 MB/sec execute 430 sec latency 49.626 ms 2024-03-11T22:33:54.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:54 smithi073 ceph-mon[66481]: pgmap v156: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 486 op/s; 350 KiB/s, 2 objects/s recovering 2024-03-11T22:33:54.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:54.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:54.814 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:54.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:54 smithi099 ceph-mon[53203]: pgmap v156: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 486 op/s; 350 KiB/s, 2 objects/s recovering 2024-03-11T22:33:54.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:54.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:54.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:33:55.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 240877 10.35 MB/sec execute 431 sec latency 45.345 ms 2024-03-11T22:33:56.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 241171 10.35 MB/sec execute 432 sec latency 58.212 ms 2024-03-11T22:33:56.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:56 smithi073 ceph-mon[66481]: pgmap v157: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 639 op/s; 350 KiB/s, 2 objects/s recovering 2024-03-11T22:33:56.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:56 smithi099 ceph-mon[53203]: pgmap v157: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 639 op/s; 350 KiB/s, 2 objects/s recovering 2024-03-11T22:33:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:57.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 241548 10.35 MB/sec execute 433 sec latency 42.808 ms 2024-03-11T22:33:57.907 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:57 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:57.907 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:57 smithi073 ceph-mon[66481]: Upgrade: osd.3 is safe to restart 2024-03-11T22:33:57.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:57 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-11T22:33:57.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:57 smithi099 ceph-mon[53203]: Upgrade: osd.3 is safe to restart 2024-03-11T22:33:58.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 241823 10.34 MB/sec execute 434 sec latency 52.093 ms 2024-03-11T22:33:58.913 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:33:58 smithi073 ceph-mon[66481]: pgmap v158: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 472 op/s 2024-03-11T22:33:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:33:58 smithi099 ceph-mon[53203]: pgmap v158: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 472 op/s 2024-03-11T22:33:59.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 242050 10.34 MB/sec execute 435 sec latency 61.123 ms 2024-03-11T22:34:00.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 242265 10.34 MB/sec execute 436 sec latency 65.378 ms 2024-03-11T22:34:00.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: Upgrade: Updating osd.3 2024-03-11T22:34:00.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:00.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:34:00.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:00.770 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: Deploying daemon osd.3 on smithi073 2024-03-11T22:34:00.771 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:00 smithi073 ceph-mon[66481]: pgmap v159: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.6 MiB/s wr, 505 op/s 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: Upgrade: Updating osd.3 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: Deploying daemon osd.3 on smithi073 2024-03-11T22:34:00.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:00 smithi099 ceph-mon[53203]: pgmap v159: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.6 MiB/s wr, 505 op/s 2024-03-11T22:34:01.664 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:34:01.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 242453 10.32 MB/sec execute 437 sec latency 95.286 ms 2024-03-11T22:34:01.829 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:01 smithi073 ceph-mon[66481]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:01.829 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:01 smithi073 ceph-mon[66481]: pgmap v160: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 462 op/s 2024-03-11T22:34:02.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:01 smithi099 ceph-mon[53203]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:02.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:01 smithi099 ceph-mon[53203]: pgmap v160: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 462 op/s 2024-03-11T22:34:02.463 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:34:02.463 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (6m) 27s ago 15m 15.6M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (2m) 27s ago 15m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (2m) 2m ago 15m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (6m) 27s ago 15m 44.2M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (9m) 27s ago 9m 38.1M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (9m) 2m ago 9m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (5m) 27s ago 17m 502M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (4m) 2m ago 15m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (4m) 27s ago 17m 47.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (3m) 2m ago 15m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (9m) 27s ago 9m 276M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (9m) 2m ago 9m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 27s ago 15m 9537k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (5m) 2m ago 14m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (2m) 27s ago 14m 302M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (87s) 27s ago 13m 240M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:34:02.464 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (37s) 27s ago 13m 14.7M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (12m) 27s ago 12m 440M 1936M 17.2.0 e1d6a67b021e 0c8cbab221c7 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (12m) 2m ago 12m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (12m) 2m ago 12m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (11m) 2m ago 11m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (11m) 2m ago 11m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:34:02.465 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (4m) 27s ago 14m 61.0M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:34:02.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 242738 10.31 MB/sec execute 438 sec latency 68.459 ms 2024-03-11T22:34:02.851 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:02 smithi073 ceph-mon[66481]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:02.851 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:02 smithi073 ceph-mon[66481]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:34:03.067 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 3 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 7 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:34:03.068 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:34:03.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:02 smithi099 ceph-mon[53203]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:03.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:02 smithi099 ceph-mon[53203]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "9/23 daemons upgraded", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:34:03.674 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:34:03.675 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:34:03.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 242947 10.30 MB/sec execute 439 sec latency 56.609 ms 2024-03-11T22:34:03.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:03 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/3729005608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:03.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:03 smithi073 ceph-mon[66481]: pgmap v161: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.6 MiB/s wr, 351 op/s 2024-03-11T22:34:03.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:03 smithi073 ceph-mon[66481]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:04.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:03 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/3729005608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:04.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:03 smithi099 ceph-mon[53203]: pgmap v161: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.6 MiB/s wr, 351 op/s 2024-03-11T22:34:04.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:03 smithi099 ceph-mon[53203]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:04.269 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:34:04.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 243125 10.28 MB/sec execute 440 sec latency 66.760 ms 2024-03-11T22:34:04.933 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:04 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/3297096576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:34:05.192 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:05 smithi073 systemd[1]: Stopping Ceph osd.3 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:34:05.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:04 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/3297096576' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:34:05.698 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:05 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[46258]: 2024-03-11T22:34:05.497+0000 7fa2f78e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:05.698 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:05 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[46258]: 2024-03-11T22:34:05.497+0000 7fa2f78e1700 -1 osd.3 107 *** Got signal Terminated *** 2024-03-11T22:34:05.698 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:05 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[46258]: 2024-03-11T22:34:05.497+0000 7fa2f78e1700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:05.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 243408 10.28 MB/sec execute 441 sec latency 52.370 ms 2024-03-11T22:34:05.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:05 smithi073 ceph-mon[66481]: osd.3 marked itself down and dead 2024-03-11T22:34:05.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:05 smithi073 ceph-mon[66481]: pgmap v162: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 448 op/s 2024-03-11T22:34:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:05 smithi099 ceph-mon[53203]: osd.3 marked itself down and dead 2024-03-11T22:34:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:05 smithi099 ceph-mon[53203]: pgmap v162: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 448 op/s 2024-03-11T22:34:06.722 INFO:teuthology.orchestra.run.smithi073.stdout: 5 243437 10.26 MB/sec execute 442 sec latency 946.126 ms 2024-03-11T22:34:06.758 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:06 smithi073 podman[80054]: 2024-03-11 22:34:06.494452604 +0000 UTC m=+1.143237217 container died 0c8cbab221c71887f408eb3cdfb500eee0febd585c79f0ccc2456e2df65010d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=754, vcs-type=git, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T22:34:07.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:34:07.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: osdmap e108: 8 total, 7 up, 8 in 2024-03-11T22:34:07.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:06 smithi073 ceph-mon[66481]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.011 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:06 smithi073 podman[80054]: 2024-03-11 22:34:06.961005008 +0000 UTC m=+1.609789627 container cleanup 0c8cbab221c71887f408eb3cdfb500eee0febd585c79f0ccc2456e2df65010d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, architecture=x86_64, ceph=True, vendor=Red Hat, Inc., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-type=git, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-03-11T22:34:07.011 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:06 smithi073 bash[80054]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: osdmap e108: 8 total, 7 up, 8 in 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:06 smithi099 ceph-mon[53203]: failed to encode map e108 with expected crc 2024-03-11T22:34:07.513 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:07 smithi073 podman[80070]: 2024-03-11 22:34:07.288221643 +0000 UTC m=+0.793525971 container remove 0c8cbab221c71887f408eb3cdfb500eee0febd585c79f0ccc2456e2df65010d8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, version=8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git) 2024-03-11T22:34:07.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 243843 10.26 MB/sec execute 443 sec latency 1069.827 ms 2024-03-11T22:34:07.764 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:07 smithi073 podman[80185]: 2024-03-11 22:34:07.652068991 +0000 UTC m=+0.048359581 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:08.015 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:07 smithi073 podman[80185]: 2024-03-11 22:34:07.786193585 +0000 UTC m=+0.182484203 container create 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: osdmap e109: 8 total, 7 up, 8 in 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.017 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.017 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.017 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:07 smithi073 ceph-mon[66481]: pgmap v165: 97 pgs: 16 stale+active+clean, 81 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.7 MiB/s wr, 427 op/s 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: osdmap e109: 8 total, 7 up, 8 in 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: failed to encode map e109 with expected crc 2024-03-11T22:34:08.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:07 smithi099 ceph-mon[53203]: pgmap v165: 97 pgs: 16 stale+active+clean, 81 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.7 MiB/s wr, 427 op/s 2024-03-11T22:34:08.266 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:08 smithi073 podman[80185]: 2024-03-11 22:34:08.185798623 +0000 UTC m=+0.582089201 container init 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-11T22:34:08.266 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:08 smithi073 podman[80185]: 2024-03-11 22:34:08.193142885 +0000 UTC m=+0.589433463 container start 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:34:08.517 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:08 smithi073 podman[80185]: 2024-03-11 22:34:08.353761168 +0000 UTC m=+0.750051744 container attach 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T22:34:08.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 244409 10.26 MB/sec execute 444 sec latency 26.756 ms 2024-03-11T22:34:08.772 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:08 smithi073 podman[80185]: 2024-03-11 22:34:08.65602139 +0000 UTC m=+1.052311961 container died 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T22:34:09.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 245002 10.28 MB/sec execute 445 sec latency 35.409 ms 2024-03-11T22:34:09.779 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:09.780 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:09.780 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:09 smithi073 podman[80210]: 2024-03-11 22:34:09.603180824 +0000 UTC m=+0.940736672 container remove 8c0e73b7f3c5ee24a2ec2df8863c37a1944ffcc062256136f244bef1f8007d5a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T22:34:09.780 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:09 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-11T22:34:09.780 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:09 smithi073 systemd[1]: Stopped Ceph osd.3 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:34:09.780 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:09 smithi073 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.3.service: Consumed 5min 5.628s CPU time. 2024-03-11T22:34:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:10.035 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:09 smithi073 systemd[1]: Starting Ceph osd.3 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:34:10.287 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:10 smithi073 podman[80310]: 2024-03-11 22:34:10.137501631 +0000 UTC m=+0.030610259 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:10.287 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:10 smithi073 podman[80310]: 2024-03-11 22:34:10.281253665 +0000 UTC m=+0.174362293 container create ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T22:34:10.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 245454 10.29 MB/sec execute 446 sec latency 28.004 ms 2024-03-11T22:34:10.800 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:10 smithi073 ceph-mon[66481]: pgmap v166: 97 pgs: 11 active+undersized, 16 active+undersized+degraded, 5 stale+active+clean, 65 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.2 MiB/s wr, 463 op/s; 142/1632 objects degraded (8.701%) 2024-03-11T22:34:10.800 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:10 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 142/1632 objects degraded (8.701%), 16 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:10.801 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:10 smithi073 podman[80310]: 2024-03-11 22:34:10.66914173 +0000 UTC m=+0.562250357 container init ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T22:34:10.801 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:10 smithi073 podman[80310]: 2024-03-11 22:34:10.675585746 +0000 UTC m=+0.568694377 container start ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:10.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:10 smithi099 ceph-mon[53203]: pgmap v166: 97 pgs: 11 active+undersized, 16 active+undersized+degraded, 5 stale+active+clean, 65 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.2 MiB/s wr, 463 op/s; 142/1632 objects degraded (8.701%) 2024-03-11T22:34:10.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:10 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 142/1632 objects degraded (8.701%), 16 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:11.051 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:10 smithi073 podman[80310]: 2024-03-11 22:34:10.820614858 +0000 UTC m=+0.713723494 container attach ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 bash[80310]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T22:34:11.304 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T22:34:11.305 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate[80324]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T22:34:11.557 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:11 smithi073 podman[80310]: 2024-03-11 22:34:11.299421896 +0000 UTC m=+1.192530528 container died ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T22:34:11.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 245799 10.30 MB/sec execute 447 sec latency 51.644 ms 2024-03-11T22:34:11.809 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:11 smithi073 ceph-mon[66481]: pgmap v167: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.9 MiB/s wr, 528 op/s; 233/1713 objects degraded (13.602%); 668 B/s, 0 objects/s recovering 2024-03-11T22:34:12.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:11 smithi099 ceph-mon[53203]: pgmap v167: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.9 MiB/s wr, 528 op/s; 233/1713 objects degraded (13.602%); 668 B/s, 0 objects/s recovering 2024-03-11T22:34:12.569 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:12 smithi073 podman[80445]: 2024-03-11 22:34:12.548897524 +0000 UTC m=+1.242014026 container remove ff02885638343de121dc632d742b1d576d04f5e003d6d2865e682d9cdaedd64f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T22:34:12.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 246171 10.29 MB/sec execute 448 sec latency 38.263 ms 2024-03-11T22:34:12.820 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:12 smithi073 podman[80496]: 2024-03-11 22:34:12.713941798 +0000 UTC m=+0.025309504 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:13.098 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:12 smithi073 podman[80496]: 2024-03-11 22:34:12.841737024 +0000 UTC m=+0.153104713 container create 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:34:13.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 246523 10.28 MB/sec execute 449 sec latency 88.317 ms 2024-03-11T22:34:13.855 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:13 smithi073 ceph-mon[66481]: pgmap v168: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s wr, 383 op/s; 233/1713 objects degraded (13.602%); 668 B/s, 0 objects/s recovering 2024-03-11T22:34:13.856 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:13 smithi073 podman[80496]: 2024-03-11 22:34:13.677191201 +0000 UTC m=+0.988558910 container init 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T22:34:13.856 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:13 smithi073 podman[80496]: 2024-03-11 22:34:13.682992067 +0000 UTC m=+0.994359767 container start 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-11T22:34:13.856 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:13 smithi073 ceph-osd[80514]: -- 172.21.15.73:0/135103126 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ec3744f1e0 con 0x55ec37420c00 2024-03-11T22:34:13.856 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:13 smithi073 bash[80496]: 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 2024-03-11T22:34:13.856 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:13 smithi073 systemd[1]: Started Ceph osd.3 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:34:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:13 smithi099 ceph-mon[53203]: pgmap v168: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.1 MiB/s wr, 383 op/s; 233/1713 objects degraded (13.602%); 668 B/s, 0 objects/s recovering 2024-03-11T22:34:14.369 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:14 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:34:14.264+0000 7f4225d577c0 -1 Falling back to public interface 2024-03-11T22:34:14.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 246781 10.27 MB/sec execute 450 sec latency 49.118 ms 2024-03-11T22:34:14.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:14 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:14.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:14 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:14.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:14 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:14 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:14 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:14 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:15.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 247143 10.27 MB/sec execute 451 sec latency 76.119 ms 2024-03-11T22:34:16.182 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:15 smithi073 ceph-mon[66481]: pgmap v169: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 352 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 498 op/s; 231/1704 objects degraded (13.556%); 542 B/s, 0 objects/s recovering 2024-03-11T22:34:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:15 smithi099 ceph-mon[53203]: pgmap v169: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 352 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 498 op/s; 231/1704 objects degraded (13.556%); 542 B/s, 0 objects/s recovering 2024-03-11T22:34:16.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 247411 10.26 MB/sec execute 452 sec latency 88.347 ms 2024-03-11T22:34:16.979 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:16 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 231/1704 objects degraded (13.556%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:17.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:16 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 231/1704 objects degraded (13.556%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:17.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 247768 10.25 MB/sec execute 453 sec latency 52.929 ms 2024-03-11T22:34:18.018 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:17 smithi073 ceph-mon[66481]: pgmap v170: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.9 MiB/s wr, 462 op/s; 232/1710 objects degraded (13.567%); 492 B/s, 0 objects/s recovering 2024-03-11T22:34:18.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:17 smithi099 ceph-mon[53203]: pgmap v170: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 348 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.9 MiB/s wr, 462 op/s; 232/1710 objects degraded (13.567%); 492 B/s, 0 objects/s recovering 2024-03-11T22:34:18.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 247992 10.24 MB/sec execute 454 sec latency 49.583 ms 2024-03-11T22:34:19.730 INFO:teuthology.orchestra.run.smithi073.stdout: 5 248227 10.24 MB/sec execute 455 sec latency 66.693 ms 2024-03-11T22:34:19.804 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:19 smithi073 ceph-mon[66481]: pgmap v171: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 584 op/s; 208/1575 objects degraded (13.206%); 445 B/s, 0 objects/s recovering 2024-03-11T22:34:20.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:19 smithi099 ceph-mon[53203]: pgmap v171: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 584 op/s; 208/1575 objects degraded (13.206%); 445 B/s, 0 objects/s recovering 2024-03-11T22:34:20.735 INFO:teuthology.orchestra.run.smithi073.stdout: 5 248643 10.25 MB/sec execute 456 sec latency 53.737 ms 2024-03-11T22:34:20.861 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:20 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:34:20.829+0000 7f4225d577c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-11T22:34:21.735 INFO:teuthology.orchestra.run.smithi073.stdout: 5 248959 10.24 MB/sec execute 457 sec latency 62.082 ms 2024-03-11T22:34:21.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:21 smithi073 ceph-mon[66481]: pgmap v172: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.7 MiB/s wr, 465 op/s; 206/1551 objects degraded (13.282%); 445 B/s, 0 objects/s recovering 2024-03-11T22:34:21.929 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:21 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 206/1551 objects degraded (13.282%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:22.187 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:34:22.142+0000 7f4225d577c0 -1 osd.3 107 log_to_monitors true 2024-03-11T22:34:22.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:21 smithi099 ceph-mon[53203]: pgmap v172: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.7 MiB/s wr, 465 op/s; 206/1551 objects degraded (13.282%); 445 B/s, 0 objects/s recovering 2024-03-11T22:34:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:21 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 206/1551 objects degraded (13.282%), 26 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:22.735 INFO:teuthology.orchestra.run.smithi073.stdout: 5 249248 10.25 MB/sec execute 458 sec latency 45.113 ms 2024-03-11T22:34:22.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:22 smithi073 ceph-mon[66481]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:34:22.961 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:34:22 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:34:22.816+0000 7f42171af700 -1 osd.3 107 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:34:23.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:22 smithi099 ceph-mon[53203]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:34:23.735 INFO:teuthology.orchestra.run.smithi073.stdout: 5 249520 10.24 MB/sec execute 459 sec latency 43.086 ms 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: failed to encode map e110 with expected crc 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: osdmap e110: 8 total, 7 up, 8 in 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: pgmap v174: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 505 op/s; 206/1551 objects degraded (13.282%) 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: failed to encode map e110 with expected crc 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: osdmap e110: 8 total, 7 up, 8 in 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: pgmap v174: 97 pgs: 15 active+undersized, 26 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 505 op/s; 206/1551 objects degraded (13.282%) 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:24.735 INFO:teuthology.orchestra.run.smithi073.stdout: 5 249533 10.22 MB/sec execute 460 sec latency 967.364 ms 2024-03-11T22:34:25.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:34:25.058 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: failed to encode map e111 with expected crc 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: failed to encode map e111 with expected crc 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702] boot 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:25.059 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: failed to encode map e111 with expected crc 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: failed to encode map e111 with expected crc 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: osd.3 [v2:172.21.15.73:6826/2781379702,v1:172.21.15.73:6827/2781379702] boot 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:25.741 INFO:teuthology.orchestra.run.smithi073.stdout: 5 249672 10.20 MB/sec execute 461 sec latency 1153.657 ms 2024-03-11T22:34:25.838 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T22:34:25.838 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: failed to encode map e112 with expected crc 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: pgmap v177: 97 pgs: 14 peering, 8 active+undersized, 19 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 334 op/s; 163/1653 objects degraded (9.861%) 2024-03-11T22:34:25.839 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:25 smithi073 ceph-mon[66481]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: failed to encode map e112 with expected crc 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: pgmap v177: 97 pgs: 14 peering, 8 active+undersized, 19 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 334 op/s; 163/1653 objects degraded (9.861%) 2024-03-11T22:34:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:25 smithi099 ceph-mon[53203]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T22:34:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: 5 249961 10.19 MB/sec execute 462 sec latency 957.552 ms 2024-03-11T22:34:26.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:26 smithi073 ceph-mon[66481]: failed to encode map e113 with expected crc 2024-03-11T22:34:26.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:26 smithi073 ceph-mon[66481]: failed to encode map e113 with expected crc 2024-03-11T22:34:26.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:26 smithi073 ceph-mon[66481]: failed to encode map e113 with expected crc 2024-03-11T22:34:26.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:26 smithi073 ceph-mon[66481]: failed to encode map e113 with expected crc 2024-03-11T22:34:26.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:26 smithi073 ceph-mon[66481]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:26 smithi099 ceph-mon[53203]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:26 smithi099 ceph-mon[53203]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:26 smithi099 ceph-mon[53203]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:26 smithi099 ceph-mon[53203]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:26 smithi099 ceph-mon[53203]: failed to encode map e113 with expected crc 2024-03-11T22:34:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 250214 10.19 MB/sec execute 463 sec latency 64.474 ms 2024-03-11T22:34:27.910 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:27 smithi073 ceph-mon[66481]: failed to encode map e114 with expected crc 2024-03-11T22:34:27.911 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:27 smithi073 ceph-mon[66481]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T22:34:27.911 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:27 smithi073 ceph-mon[66481]: failed to encode map e114 with expected crc 2024-03-11T22:34:27.911 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:27 smithi073 ceph-mon[66481]: failed to encode map e114 with expected crc 2024-03-11T22:34:27.911 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:27 smithi073 ceph-mon[66481]: pgmap v180: 97 pgs: 1 remapped+peering, 14 peering, 7 active+undersized, 15 active+undersized+degraded, 60 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 449 op/s; 136/1623 objects degraded (8.380%); 40 KiB/s, 4 objects/s recovering 2024-03-11T22:34:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:27 smithi099 ceph-mon[53203]: failed to encode map e114 with expected crc 2024-03-11T22:34:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:27 smithi099 ceph-mon[53203]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T22:34:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:27 smithi099 ceph-mon[53203]: failed to encode map e114 with expected crc 2024-03-11T22:34:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:27 smithi099 ceph-mon[53203]: failed to encode map e114 with expected crc 2024-03-11T22:34:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:27 smithi099 ceph-mon[53203]: pgmap v180: 97 pgs: 1 remapped+peering, 14 peering, 7 active+undersized, 15 active+undersized+degraded, 60 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.7 MiB/s wr, 449 op/s; 136/1623 objects degraded (8.380%); 40 KiB/s, 4 objects/s recovering 2024-03-11T22:34:28.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 250474 10.18 MB/sec execute 464 sec latency 469.046 ms 2024-03-11T22:34:28.922 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 136/1623 objects degraded (8.380%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:28.923 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:28 smithi073 ceph-mon[66481]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 136/1623 objects degraded (8.380%), 15 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:28 smithi099 ceph-mon[53203]: failed to encode map e115 with expected crc 2024-03-11T22:34:29.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 250727 10.17 MB/sec execute 465 sec latency 853.847 ms 2024-03-11T22:34:29.960 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: failed to encode map e116 with expected crc 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: failed to encode map e116 with expected crc 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: failed to encode map e116 with expected crc 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: failed to encode map e116 with expected crc 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: pgmap v183: 97 pgs: 1 remapped+peering, 5 peering, 1 active+undersized+remapped, 2 active+undersized+degraded, 88 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.2 MiB/s wr, 406 op/s; 21/1554 objects degraded (1.351%); 2.7 MiB/s, 47 objects/s recovering 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:29.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:29.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:29.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:29.962 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: failed to encode map e116 with expected crc 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: failed to encode map e116 with expected crc 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: failed to encode map e116 with expected crc 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: failed to encode map e116 with expected crc 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: pgmap v183: 97 pgs: 1 remapped+peering, 5 peering, 1 active+undersized+remapped, 2 active+undersized+degraded, 88 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.2 MiB/s wr, 406 op/s; 21/1554 objects degraded (1.351%); 2.7 MiB/s, 47 objects/s recovering 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:30.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:30.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 251037 10.16 MB/sec execute 466 sec latency 968.848 ms 2024-03-11T22:34:31.006 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:31.006 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-11T22:34:31.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T22:34:31.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.007 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:30 smithi073 ceph-mon[66481]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:30 smithi099 ceph-mon[53203]: failed to encode map e117 with expected crc 2024-03-11T22:34:31.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 251306 10.16 MB/sec execute 467 sec latency 57.090 ms 2024-03-11T22:34:32.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:31 smithi073 ceph-mon[66481]: pgmap v185: 97 pgs: 1 peering, 1 active+undersized+remapped, 2 active+undersized+degraded, 93 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.0 MiB/s wr, 364 op/s; 21/1578 objects degraded (1.331%); 2.4 MiB/s, 44 objects/s recovering 2024-03-11T22:34:32.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:31 smithi099 ceph-mon[53203]: pgmap v185: 97 pgs: 1 peering, 1 active+undersized+remapped, 2 active+undersized+degraded, 93 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.0 MiB/s wr, 364 op/s; 21/1578 objects degraded (1.331%); 2.4 MiB/s, 44 objects/s recovering 2024-03-11T22:34:32.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 251599 10.15 MB/sec execute 468 sec latency 78.035 ms 2024-03-11T22:34:33.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:32 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:33.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:32 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:34:33.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 251933 10.15 MB/sec execute 469 sec latency 56.340 ms 2024-03-11T22:34:34.105 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:33 smithi073 ceph-mon[66481]: pgmap v186: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 337 op/s; 2.3 MiB/s, 41 objects/s recovering 2024-03-11T22:34:34.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:33 smithi099 ceph-mon[53203]: pgmap v186: 97 pgs: 1 peering, 96 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 337 op/s; 2.3 MiB/s, 41 objects/s recovering 2024-03-11T22:34:34.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 252190 10.14 MB/sec execute 470 sec latency 65.701 ms 2024-03-11T22:34:34.786 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:34:35.132 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:34 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded) 2024-03-11T22:34:35.132 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:34 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:34:35.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:34 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded) 2024-03-11T22:34:35.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:34 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:34:35.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 252382 10.14 MB/sec execute 471 sec latency 59.383 ms 2024-03-11T22:34:35.817 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:34:35.817 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (7m) 12s ago 16m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:34:35.817 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (3m) 12s ago 16m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:34:35.817 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (3m) 3m ago 15m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (6m) 12s ago 16m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (10m) 12s ago 10m 38.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (10m) 3m ago 10m 13.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (6m) 12s ago 17m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (5m) 3m ago 15m 448M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (4m) 12s ago 17m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (4m) 3m ago 15m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (10m) 12s ago 10m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (9m) 3m ago 9m 83.8M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (6m) 12s ago 16m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (6m) 3m ago 15m 16.8M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (2m) 12s ago 14m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (2m) 12s ago 14m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:34:35.818 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (71s) 12s ago 13m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (23s) 12s ago 13m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (13m) 3m ago 13m 555M 2704M 17.2.0 e1d6a67b021e 05253cc2b1df 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (12m) 3m ago 12m 414M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (12m) 3m ago 12m 366M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (11m) 3m ago 11m 533M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:34:35.819 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (5m) 12s ago 15m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:34:36.166 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:35 smithi073 ceph-mon[66481]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:36.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:35 smithi073 ceph-mon[66481]: from='client.44253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:36.167 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:35 smithi073 ceph-mon[66481]: pgmap v187: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 482 op/s; 1.7 MiB/s, 25 objects/s recovering 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-11T22:34:36.408 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 4 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:34:36.409 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:34:36.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:35 smithi099 ceph-mon[53203]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:36.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:35 smithi099 ceph-mon[53203]: from='client.44253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:36.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:35 smithi099 ceph-mon[53203]: pgmap v187: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 482 op/s; 1.7 MiB/s, 25 objects/s recovering 2024-03-11T22:34:36.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 252626 10.14 MB/sec execute 472 sec latency 61.303 ms 2024-03-11T22:34:36.946 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:34:36.946 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:34:36.946 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "10/23 daemons upgraded", 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:34:36.947 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:34:37.205 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:36 smithi073 ceph-mon[66481]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:37.205 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:36 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2343437840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:37.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:36 smithi099 ceph-mon[53203]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:37.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:36 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2343437840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:34:37.578 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:34:37.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 252877 10.13 MB/sec execute 473 sec latency 48.227 ms 2024-03-11T22:34:38.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:38 smithi073 ceph-mon[66481]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:38.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:38 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/324172037' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:34:38.214 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:38 smithi073 ceph-mon[66481]: pgmap v188: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 424 op/s; 1.3 MiB/s, 12 objects/s recovering 2024-03-11T22:34:38.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:38 smithi099 ceph-mon[53203]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:34:38.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:38 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/324172037' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:34:38.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:38 smithi099 ceph-mon[53203]: pgmap v188: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 424 op/s; 1.3 MiB/s, 12 objects/s recovering 2024-03-11T22:34:38.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 253433 10.14 MB/sec execute 474 sec latency 28.036 ms 2024-03-11T22:34:39.727 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:39.727 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:39.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 253846 10.13 MB/sec execute 475 sec latency 36.054 ms 2024-03-11T22:34:39.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:39.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:40.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 254219 10.13 MB/sec execute 476 sec latency 45.200 ms 2024-03-11T22:34:40.771 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:40 smithi073 ceph-mon[66481]: pgmap v189: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.0 MiB/s wr, 462 op/s; 292 KiB/s, 8 objects/s recovering 2024-03-11T22:34:40.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:40 smithi099 ceph-mon[53203]: pgmap v189: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.0 MiB/s wr, 462 op/s; 292 KiB/s, 8 objects/s recovering 2024-03-11T22:34:41.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 254576 10.12 MB/sec execute 477 sec latency 46.738 ms 2024-03-11T22:34:42.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 254933 10.13 MB/sec execute 478 sec latency 43.934 ms 2024-03-11T22:34:42.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:42 smithi073 ceph-mon[66481]: pgmap v190: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.5 MiB/s wr, 436 op/s; 247 KiB/s, 3 objects/s recovering 2024-03-11T22:34:42.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:42 smithi099 ceph-mon[53203]: pgmap v190: 97 pgs: 97 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.5 MiB/s wr, 436 op/s; 247 KiB/s, 3 objects/s recovering 2024-03-11T22:34:43.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 255289 10.12 MB/sec execute 479 sec latency 68.733 ms 2024-03-11T22:34:44.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 255617 10.12 MB/sec execute 480 sec latency 47.229 ms 2024-03-11T22:34:44.840 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:44 smithi073 ceph-mon[66481]: pgmap v191: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.6 MiB/s wr, 442 op/s; 183 KiB/s, 2 objects/s recovering 2024-03-11T22:34:44.841 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:44 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:44 smithi099 ceph-mon[53203]: pgmap v191: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.6 MiB/s wr, 442 op/s; 183 KiB/s, 2 objects/s recovering 2024-03-11T22:34:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:44 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:45.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 255878 10.12 MB/sec execute 481 sec latency 63.487 ms 2024-03-11T22:34:45.878 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:45 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:45.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:45 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:46.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 256189 10.12 MB/sec execute 482 sec latency 56.748 ms 2024-03-11T22:34:46.891 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: Upgrade: osd.4 is safe to restart 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: pgmap v192: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 517 op/s 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: Upgrade: Updating osd.4 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:46.892 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:46 smithi073 ceph-mon[66481]: Deploying daemon osd.4 on smithi099 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: Upgrade: osd.4 is safe to restart 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: pgmap v192: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 517 op/s 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: Upgrade: Updating osd.4 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:34:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:46 smithi099 ceph-mon[53203]: Deploying daemon osd.4 on smithi099 2024-03-11T22:34:47.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 256445 10.12 MB/sec execute 483 sec latency 58.311 ms 2024-03-11T22:34:47.899 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:47 smithi073 ceph-mon[66481]: pgmap v193: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 433 op/s 2024-03-11T22:34:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:47 smithi099 ceph-mon[53203]: pgmap v193: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 433 op/s 2024-03-11T22:34:48.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 256770 10.12 MB/sec execute 484 sec latency 72.030 ms 2024-03-11T22:34:49.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 257022 10.10 MB/sec execute 485 sec latency 56.578 ms 2024-03-11T22:34:49.959 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:49 smithi073 ceph-mon[66481]: pgmap v194: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-03-11T22:34:49.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:49 smithi099 ceph-mon[53203]: pgmap v194: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-03-11T22:34:50.742 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:50 smithi099 systemd[1]: Stopping Ceph osd.4 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:34:50.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 257355 10.10 MB/sec execute 486 sec latency 56.610 ms 2024-03-11T22:34:51.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:50 smithi073 ceph-mon[66481]: osd.4 marked itself down and dead 2024-03-11T22:34:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:50 smithi099 ceph-mon[53203]: osd.4 marked itself down and dead 2024-03-11T22:34:51.242 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:50 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[31406]: 2024-03-11T22:34:50.738+0000 7fbb7d007700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:51.242 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:50 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[31406]: 2024-03-11T22:34:50.738+0000 7fbb7d007700 -1 osd.4 117 *** Got signal Terminated *** 2024-03-11T22:34:51.243 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:50 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[31406]: 2024-03-11T22:34:50.738+0000 7fbb7d007700 -1 osd.4 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:51.703 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:51 smithi099 podman[57689]: 2024-03-11 22:34:51.392215864 +0000 UTC m=+0.794086597 container died 05253cc2b1df3b7f4a39eeead06f051fc67391e0fafdf7c4ea5463d6c4b0be06 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, com.redhat.component=centos-stream-container, version=8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870) 2024-03-11T22:34:51.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 257367 10.08 MB/sec execute 487 sec latency 974.829 ms 2024-03-11T22:34:51.992 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:51 smithi099 podman[57689]: 2024-03-11 22:34:51.796639448 +0000 UTC m=+1.198510192 container cleanup 05253cc2b1df3b7f4a39eeead06f051fc67391e0fafdf7c4ea5463d6c4b0be06 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, GIT_BRANCH=HEAD, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, ceph=True, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.buildah.version=1.19.8) 2024-03-11T22:34:51.992 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:51 smithi099 bash[57689]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4 2024-03-11T22:34:51.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:51 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:34:51.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:51 smithi099 ceph-mon[53203]: failed to encode map e118 with expected crc 2024-03-11T22:34:51.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:51 smithi099 ceph-mon[53203]: osdmap e118: 8 total, 7 up, 8 in 2024-03-11T22:34:51.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:51 smithi099 ceph-mon[53203]: pgmap v196: 97 pgs: 14 stale+active+clean, 83 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-03-11T22:34:52.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:51 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:34:52.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:51 smithi073 ceph-mon[66481]: failed to encode map e118 with expected crc 2024-03-11T22:34:52.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:51 smithi073 ceph-mon[66481]: osdmap e118: 8 total, 7 up, 8 in 2024-03-11T22:34:52.016 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:51 smithi073 ceph-mon[66481]: pgmap v196: 97 pgs: 14 stale+active+clean, 83 active+clean; 348 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-03-11T22:34:52.464 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57704]: 2024-03-11 22:34:52.023382389 +0000 UTC m=+0.624596975 container remove 05253cc2b1df3b7f4a39eeead06f051fc67391e0fafdf7c4ea5463d6c4b0be06 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-11T22:34:52.742 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57796]: 2024-03-11 22:34:52.361134951 +0000 UTC m=+0.017964862 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:52.742 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57796]: 2024-03-11 22:34:52.496082869 +0000 UTC m=+0.152912814 container create 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:34:52.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 257770 10.08 MB/sec execute 488 sec latency 1094.544 ms 2024-03-11T22:34:52.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:52 smithi073 ceph-mon[66481]: osdmap e119: 8 total, 7 up, 8 in 2024-03-11T22:34:52.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:52 smithi073 ceph-mon[66481]: failed to encode map e119 with expected crc 2024-03-11T22:34:52.781 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:52 smithi073 ceph-mon[66481]: failed to encode map e119 with expected crc 2024-03-11T22:34:52.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:52 smithi073 ceph-mon[66481]: failed to encode map e119 with expected crc 2024-03-11T22:34:52.782 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:52 smithi073 ceph-mon[66481]: failed to encode map e119 with expected crc 2024-03-11T22:34:53.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:52 smithi099 ceph-mon[53203]: osdmap e119: 8 total, 7 up, 8 in 2024-03-11T22:34:53.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:52 smithi099 ceph-mon[53203]: failed to encode map e119 with expected crc 2024-03-11T22:34:53.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:52 smithi099 ceph-mon[53203]: failed to encode map e119 with expected crc 2024-03-11T22:34:53.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:52 smithi099 ceph-mon[53203]: failed to encode map e119 with expected crc 2024-03-11T22:34:53.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:52 smithi099 ceph-mon[53203]: failed to encode map e119 with expected crc 2024-03-11T22:34:53.065 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57796]: 2024-03-11 22:34:52.744306878 +0000 UTC m=+0.401136797 container init 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T22:34:53.065 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57796]: 2024-03-11 22:34:52.748460754 +0000 UTC m=+0.405290658 container start 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:34:53.065 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:52 smithi099 podman[57796]: 2024-03-11 22:34:52.835783227 +0000 UTC m=+0.492613133 container attach 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:34:53.492 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 podman[57796]: 2024-03-11 22:34:53.058540065 +0000 UTC m=+0.715369970 container died 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:53.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 258207 10.07 MB/sec execute 489 sec latency 35.399 ms 2024-03-11T22:34:54.225 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 podman[57822]: 2024-03-11 22:34:53.76577488 +0000 UTC m=+0.701553894 container remove 181134b09ce8a588552a1f8632f4c046818df220d319bb0850e23cf1af11edd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:54.226 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-11T22:34:54.226 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 systemd[1]: Stopped Ceph osd.4 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:34:54.226 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.4.service: Consumed 4min 48.548s CPU time. 2024-03-11T22:34:54.226 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:53 smithi099 systemd[1]: Starting Ceph osd.4 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:34:54.492 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:54 smithi099 podman[57920]: 2024-03-11 22:34:54.123553187 +0000 UTC m=+0.019392825 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:54.492 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:54 smithi099 podman[57920]: 2024-03-11 22:34:54.258300228 +0000 UTC m=+0.154139863 container create dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:54.738 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:54 smithi073 ceph-mon[66481]: pgmap v198: 97 pgs: 4 active+undersized, 1 active+undersized+degraded, 11 stale+active+clean, 81 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 392 op/s; 15/1677 objects degraded (0.894%) 2024-03-11T22:34:54.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:54.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:54.739 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:54 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 15/1677 objects degraded (0.894%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:34:54.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 258809 10.09 MB/sec execute 490 sec latency 30.007 ms 2024-03-11T22:34:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:54 smithi099 ceph-mon[53203]: pgmap v198: 97 pgs: 4 active+undersized, 1 active+undersized+degraded, 11 stale+active+clean, 81 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 392 op/s; 15/1677 objects degraded (0.894%) 2024-03-11T22:34:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:34:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:54 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 15/1677 objects degraded (0.894%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:34:54.993 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:54 smithi099 podman[57920]: 2024-03-11 22:34:54.523370672 +0000 UTC m=+0.419210309 container init dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T22:34:54.993 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:54 smithi099 podman[57920]: 2024-03-11 22:34:54.52794153 +0000 UTC m=+0.423781179 container start dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:34:54.993 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:54 smithi099 podman[57920]: 2024-03-11 22:34:54.636572251 +0000 UTC m=+0.532411902 container attach dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T22:34:55.447 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 bash[57920]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate[57936]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-11T22:34:55.448 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 podman[57920]: 2024-03-11 22:34:55.069673947 +0000 UTC m=+0.965513584 container died dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:34:55.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 259478 10.12 MB/sec execute 491 sec latency 25.196 ms 2024-03-11T22:34:56.023 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 podman[58068]: 2024-03-11 22:34:55.766468386 +0000 UTC m=+0.686872128 container remove dd8b15777c7914a11f8b71384b344533090c4026a715779d3b823944784c661e (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T22:34:56.023 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:55 smithi099 podman[58130]: 2024-03-11 22:34:55.891395236 +0000 UTC m=+0.018782554 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:56.286 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 podman[58130]: 2024-03-11 22:34:56.017506777 +0000 UTC m=+0.144894101 container create 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T22:34:56.286 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 podman[58130]: 2024-03-11 22:34:56.253882047 +0000 UTC m=+0.381269378 container init 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:34:56.286 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 podman[58130]: 2024-03-11 22:34:56.258222371 +0000 UTC m=+0.385609697 container start 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T22:34:56.704 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-osd[58149]: -- 172.21.15.99:0/2303796109 <== mon.1 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5606c46d31e0 con 0x5606c46a4c00 2024-03-11T22:34:56.705 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 bash[58130]: 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a 2024-03-11T22:34:56.705 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 systemd[1]: Started Ceph osd.4 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:34:56.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 259832 10.12 MB/sec execute 492 sec latency 60.872 ms 2024-03-11T22:34:56.752 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:56 smithi073 ceph-mon[66481]: pgmap v199: 97 pgs: 12 active+undersized, 24 active+undersized+degraded, 2 stale+active+clean, 59 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 526 op/s; 216/1650 objects degraded (13.091%) 2024-03-11T22:34:56.752 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:56.752 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:56.753 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:56.992 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:34:56.798+0000 7f57873c27c0 -1 Falling back to public interface 2024-03-11T22:34:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-mon[53203]: pgmap v199: 97 pgs: 12 active+undersized, 24 active+undersized+degraded, 2 stale+active+clean, 59 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 526 op/s; 216/1650 objects degraded (13.091%) 2024-03-11T22:34:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:34:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:34:57.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 260174 10.12 MB/sec execute 493 sec latency 74.309 ms 2024-03-11T22:34:58.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:58 smithi099 ceph-mon[53203]: pgmap v200: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.1 MiB/s wr, 284 op/s; 259/1632 objects degraded (15.870%) 2024-03-11T22:34:58.747 INFO:teuthology.orchestra.run.smithi073.stdout: 5 260456 10.11 MB/sec execute 494 sec latency 52.379 ms 2024-03-11T22:34:58.790 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:58 smithi073 ceph-mon[66481]: pgmap v200: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.1 MiB/s wr, 284 op/s; 259/1632 objects degraded (15.870%) 2024-03-11T22:34:59.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 260682 10.10 MB/sec execute 495 sec latency 48.032 ms 2024-03-11T22:34:59.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:59 smithi073 ceph-mon[66481]: pgmap v201: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.2 MiB/s wr, 468 op/s; 269/1719 objects degraded (15.649%) 2024-03-11T22:34:59.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:34:59 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 269/1719 objects degraded (15.649%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:00.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:59 smithi099 ceph-mon[53203]: pgmap v201: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.2 MiB/s wr, 468 op/s; 269/1719 objects degraded (15.649%) 2024-03-11T22:35:00.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:34:59 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 269/1719 objects degraded (15.649%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:00.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 260892 10.09 MB/sec execute 496 sec latency 76.806 ms 2024-03-11T22:35:01.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:01.444 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:01.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:01.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:01.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 261069 10.07 MB/sec execute 497 sec latency 112.374 ms 2024-03-11T22:35:02.465 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:02 smithi073 ceph-mon[66481]: pgmap v202: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.4 MiB/s wr, 543 op/s; 260/1662 objects degraded (15.644%) 2024-03-11T22:35:02.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:02 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:02.466 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:02 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:02.698 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:02 smithi099 ceph-mon[53203]: pgmap v202: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.4 MiB/s wr, 543 op/s; 260/1662 objects degraded (15.644%) 2024-03-11T22:35:02.698 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:02 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:02.699 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:02 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:02.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 261345 10.07 MB/sec execute 498 sec latency 59.927 ms 2024-03-11T22:35:03.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 261628 10.06 MB/sec execute 499 sec latency 67.280 ms 2024-03-11T22:35:03.801 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:03 smithi073 ceph-mon[66481]: pgmap v203: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 468 op/s; 260/1605 objects degraded (16.199%) 2024-03-11T22:35:03.878 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:03 smithi099 ceph-mon[53203]: pgmap v203: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 468 op/s; 260/1605 objects degraded (16.199%) 2024-03-11T22:35:04.492 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:35:04 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:35:04.374+0000 7f57873c27c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-11T22:35:04.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 261916 10.06 MB/sec execute 500 sec latency 71.904 ms 2024-03-11T22:35:05.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 262205 10.06 MB/sec execute 501 sec latency 66.824 ms 2024-03-11T22:35:06.492 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:35:06.026+0000 7f57873c27c0 -1 osd.4 117 log_to_monitors true 2024-03-11T22:35:06.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 262647 10.06 MB/sec execute 502 sec latency 55.505 ms 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: pgmap v204: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 601 op/s; 243/1527 objects degraded (15.914%) 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 243/1527 objects degraded (15.914%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:06.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:35:06.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:06 smithi073 ceph-mon[66481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:35:06.992 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:35:06.749+0000 7f577881a700 -1 osd.4 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:35:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: pgmap v204: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 601 op/s; 243/1527 objects degraded (15.914%) 2024-03-11T22:35:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 243/1527 objects degraded (15.914%), 28 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:35:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:06 smithi099 ceph-mon[53203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:35:07.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 262980 10.05 MB/sec execute 503 sec latency 35.263 ms 2024-03-11T22:35:07.897 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: osdmap e120: 8 total, 7 up, 8 in 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: from='osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: failed to encode map e120 with expected crc 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: failed to encode map e120 with expected crc 2024-03-11T22:35:07.898 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:07 smithi073 ceph-mon[66481]: failed to encode map e120 with expected crc 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: osdmap e120: 8 total, 7 up, 8 in 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: from='osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: failed to encode map e120 with expected crc 2024-03-11T22:35:07.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: failed to encode map e120 with expected crc 2024-03-11T22:35:07.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:07 smithi099 ceph-mon[53203]: failed to encode map e120 with expected crc 2024-03-11T22:35:08.058 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:35:08.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 263087 10.04 MB/sec execute 504 sec latency 997.887 ms 2024-03-11T22:35:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:35:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (7m) 45s ago 16m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:35:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (4m) 45s ago 16m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:35:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (3m) 8s ago 16m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (7m) 45s ago 16m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (10m) 45s ago 10m 38.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (10m) 8s ago 10m 14.5M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (6m) 45s ago 18m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (5m) 8s ago 16m 450M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (5m) 45s ago 18m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (4m) 8s ago 16m 41.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (10m) 45s ago 10m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (10m) 8s ago 10m 86.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (7m) 45s ago 16m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (6m) 8s ago 15m 18.4M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (3m) 45s ago 15m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (2m) 45s ago 14m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:35:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (104s) 45s ago 14m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (56s) 45s ago 13m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (12s) 8s ago 13m 11.9M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (13m) 8s ago 13m 612M 2704M 17.2.0 e1d6a67b021e 15df440335ac 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (12m) 8s ago 12m 568M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (12m) 8s ago 12m 604M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:35:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (5m) 45s ago 15m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: pgmap v206: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 530 op/s; 246/1536 objects degraded (16.016%) 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e121 with expected crc 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558] boot 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T22:35:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e121 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: pgmap v206: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 530 op/s; 246/1536 objects degraded (16.016%) 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e121 with expected crc 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: osd.4 [v2:172.21.15.99:6800/1409942558,v1:172.21.15.99:6801/1409942558] boot 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:35:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e121 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: failed to encode map e122 with expected crc 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-11T22:35:09.256 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 9 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:35:09.257 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "11/23 daemons upgraded", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:35:09.696 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:35:09.697 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:35:09.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 263276 10.04 MB/sec execute 505 sec latency 1270.587 ms 2024-03-11T22:35:09.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:09.765 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.766 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:09 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/608351176' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: failed to encode map e123 with expected crc 2024-03-11T22:35:09.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:09 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/608351176' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN Degraded data redundancy: 207/1608 objects degraded (12.873%), 15 pgs degraded 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 207/1608 objects degraded (12.873%), 15 pgs degraded 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.a is active+recovery_wait+undersized+degraded+remapped, acting [6,1] 2024-03-11T22:35:10.246 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.e is active+recovery_wait+undersized+degraded+remapped, acting [7,1] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.f is active+recovery_wait+undersized+degraded+remapped, acting [7,0] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.13 is active+recovery_wait+undersized+degraded+remapped, acting [7,2] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.14 is active+recovery_wait+undersized+degraded+remapped, acting [6,7] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.15 is active+recovery_wait+undersized+degraded+remapped, acting [7,3] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1b is active+recovering+degraded, acting [0,4,7] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-11T22:35:10.247 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-11T22:35:10.734 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:10.734 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: pgmap v210: 97 pgs: 1 active+recovering+degraded, 3 remapped+peering, 2 peering, 9 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 8 active+undersized+degraded, 67 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 385 op/s; 207/1608 objects degraded (12.873%); 356 KiB/s, 19 objects/s recovering 2024-03-11T22:35:10.734 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:10.734 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T22:35:10.735 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: failed to encode map e124 with expected crc 2024-03-11T22:35:10.735 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: failed to encode map e124 with expected crc 2024-03-11T22:35:10.735 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:10 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/948305451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:35:10.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 263528 10.03 MB/sec execute 506 sec latency 49.060 ms 2024-03-11T22:35:10.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:10.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: pgmap v210: 97 pgs: 1 active+recovering+degraded, 3 remapped+peering, 2 peering, 9 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 8 active+undersized+degraded, 67 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.7 MiB/s wr, 385 op/s; 207/1608 objects degraded (12.873%); 356 KiB/s, 19 objects/s recovering 2024-03-11T22:35:10.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:10.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T22:35:10.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: failed to encode map e124 with expected crc 2024-03-11T22:35:10.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: failed to encode map e124 with expected crc 2024-03-11T22:35:10.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:10 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/948305451' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:35:11.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 263742 10.03 MB/sec execute 507 sec latency 595.850 ms 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.064 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:11 smithi073 ceph-mon[66481]: pgmap v213: 97 pgs: 1 active+recovering+degraded, 3 remapped+peering, 2 peering, 9 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+undersized+degraded, 72 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.7 MiB/s wr, 505 op/s; 173/1635 objects degraded (10.581%); 429 KiB/s, 63 keys/s, 32 objects/s recovering 2024-03-11T22:35:12.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T22:35:12.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: failed to encode map e125 with expected crc 2024-03-11T22:35:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:11 smithi099 ceph-mon[53203]: pgmap v213: 97 pgs: 1 active+recovering+degraded, 3 remapped+peering, 2 peering, 9 active+undersized, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+undersized+degraded, 72 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.7 MiB/s wr, 505 op/s; 173/1635 objects degraded (10.581%); 429 KiB/s, 63 keys/s, 32 objects/s recovering 2024-03-11T22:35:12.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 264010 10.02 MB/sec execute 508 sec latency 828.095 ms 2024-03-11T22:35:13.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 173/1635 objects degraded (10.581%), 10 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.078 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:12 smithi073 ceph-mon[66481]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 173/1635 objects degraded (10.581%), 10 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:13.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:12 smithi099 ceph-mon[53203]: failed to encode map e126 with expected crc 2024-03-11T22:35:13.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 264326 10.01 MB/sec execute 509 sec latency 32.710 ms 2024-03-11T22:35:14.089 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.090 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:13 smithi073 ceph-mon[66481]: pgmap v216: 97 pgs: 1 active+recovering+degraded, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.6 MiB/s wr, 219 op/s; 126/1665 objects degraded (7.568%); 1.5 MiB/s, 61 keys/s, 33 objects/s recovering 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: failed to encode map e127 with expected crc 2024-03-11T22:35:14.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:13 smithi099 ceph-mon[53203]: pgmap v216: 97 pgs: 1 active+recovering+degraded, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.6 MiB/s wr, 219 op/s; 126/1665 objects degraded (7.568%); 1.5 MiB/s, 61 keys/s, 33 objects/s recovering 2024-03-11T22:35:14.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 264691 10.01 MB/sec execute 510 sec latency 62.393 ms 2024-03-11T22:35:15.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 265095 10.00 MB/sec execute 511 sec latency 58.771 ms 2024-03-11T22:35:15.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:15 smithi073 ceph-mon[66481]: pgmap v217: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.7 MiB/s wr, 239 op/s; 1.6 MiB/s, 43 keys/s, 46 objects/s recovering 2024-03-11T22:35:15.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:15 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 126/1665 objects degraded (7.568%), 7 pgs degraded) 2024-03-11T22:35:15.880 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:15 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:35:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:15 smithi099 ceph-mon[53203]: pgmap v217: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.7 MiB/s wr, 239 op/s; 1.6 MiB/s, 43 keys/s, 46 objects/s recovering 2024-03-11T22:35:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:15 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 126/1665 objects degraded (7.568%), 7 pgs degraded) 2024-03-11T22:35:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:15 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:35:16.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 265413 10.01 MB/sec execute 512 sec latency 56.949 ms 2024-03-11T22:35:17.750 INFO:teuthology.orchestra.run.smithi073.stdout: 5 265727 10.01 MB/sec execute 513 sec latency 57.044 ms 2024-03-11T22:35:17.914 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:17 smithi073 ceph-mon[66481]: pgmap v218: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 248 op/s; 1.3 MiB/s, 36 objects/s recovering 2024-03-11T22:35:18.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:17 smithi099 ceph-mon[53203]: pgmap v218: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 248 op/s; 1.3 MiB/s, 36 objects/s recovering 2024-03-11T22:35:18.748 INFO:teuthology.orchestra.run.smithi073.stdout: 5 266118 10.01 MB/sec execute 514 sec latency 61.119 ms 2024-03-11T22:35:19.752 INFO:teuthology.orchestra.run.smithi073.stdout: 5 266485 10.01 MB/sec execute 515 sec latency 41.814 ms 2024-03-11T22:35:19.949 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:19 smithi073 ceph-mon[66481]: pgmap v219: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 420 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-03-11T22:35:20.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:19 smithi099 ceph-mon[53203]: pgmap v219: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 420 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-03-11T22:35:20.752 INFO:teuthology.orchestra.run.smithi073.stdout: 5 266813 10.01 MB/sec execute 516 sec latency 33.931 ms 2024-03-11T22:35:20.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:20 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:21.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:20 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:21.752 INFO:teuthology.orchestra.run.smithi073.stdout: 5 267072 10.01 MB/sec execute 517 sec latency 59.185 ms 2024-03-11T22:35:22.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:21 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:22.010 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:21 smithi073 ceph-mon[66481]: Upgrade: osd.5 is safe to restart 2024-03-11T22:35:22.011 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:21 smithi073 ceph-mon[66481]: pgmap v220: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 472 op/s; 392 KiB/s, 18 objects/s recovering 2024-03-11T22:35:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:21 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-11T22:35:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:21 smithi099 ceph-mon[53203]: Upgrade: osd.5 is safe to restart 2024-03-11T22:35:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:21 smithi099 ceph-mon[53203]: pgmap v220: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.4 MiB/s wr, 472 op/s; 392 KiB/s, 18 objects/s recovering 2024-03-11T22:35:22.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 267413 10.01 MB/sec execute 518 sec latency 50.116 ms 2024-03-11T22:35:23.752 INFO:teuthology.orchestra.run.smithi073.stdout: 5 267669 10.00 MB/sec execute 519 sec latency 57.047 ms 2024-03-11T22:35:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: Upgrade: Updating osd.5 2024-03-11T22:35:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:23.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: Deploying daemon osd.5 on smithi099 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: pgmap v221: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 425 op/s; 293 KiB/s, 15 objects/s recovering 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:23.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: Upgrade: Updating osd.5 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: Deploying daemon osd.5 on smithi099 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: pgmap v221: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 425 op/s; 293 KiB/s, 15 objects/s recovering 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:24.077 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:24.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 267875 9.98 MB/sec execute 520 sec latency 68.842 ms 2024-03-11T22:35:25.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 268180 9.98 MB/sec execute 521 sec latency 43.848 ms 2024-03-11T22:35:25.866 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:25 smithi099 ceph-mon[53203]: pgmap v222: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.9 MiB/s wr, 488 op/s; 262 KiB/s, 13 objects/s recovering 2024-03-11T22:35:25.879 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:25 smithi073 ceph-mon[66481]: pgmap v222: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.9 MiB/s wr, 488 op/s; 262 KiB/s, 13 objects/s recovering 2024-03-11T22:35:26.648 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:26 smithi099 systemd[1]: Stopping Ceph osd.5 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:35:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: 5 268482 9.97 MB/sec execute 522 sec latency 66.225 ms 2024-03-11T22:35:26.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:26 smithi073 ceph-mon[66481]: osd.5 marked itself down and dead 2024-03-11T22:35:26.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:26 smithi099 ceph-mon[53203]: osd.5 marked itself down and dead 2024-03-11T22:35:26.992 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[34368]: 2024-03-11T22:35:26.639+0000 7f805c8db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:35:26.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[34368]: 2024-03-11T22:35:26.639+0000 7f805c8db700 -1 osd.5 127 *** Got signal Terminated *** 2024-03-11T22:35:26.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[34368]: 2024-03-11T22:35:26.639+0000 7f805c8db700 -1 osd.5 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:35:27.704 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:27 smithi099 podman[59912]: 2024-03-11 22:35:27.426443019 +0000 UTC m=+0.929624646 container died 15df440335ac83543edbe74c011f6a4af32fb3dcec9d185929709a077432dfee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-03-11T22:35:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 268499 9.95 MB/sec execute 523 sec latency 986.454 ms 2024-03-11T22:35:27.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: osdmap e128: 8 total, 7 up, 8 in 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: failed to encode map e128 with expected crc 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: failed to encode map e128 with expected crc 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: failed to encode map e128 with expected crc 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: failed to encode map e128 with expected crc 2024-03-11T22:35:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:27 smithi099 ceph-mon[53203]: pgmap v224: 97 pgs: 5 peering, 9 stale+active+clean, 83 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 568 op/s 2024-03-11T22:35:27.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:27 smithi099 podman[59912]: 2024-03-11 22:35:27.797567335 +0000 UTC m=+1.300748964 container cleanup 15df440335ac83543edbe74c011f6a4af32fb3dcec9d185929709a077432dfee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, RELEASE=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_BRANCH=HEAD) 2024-03-11T22:35:27.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:27 smithi099 bash[59912]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5 2024-03-11T22:35:28.028 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: osdmap e128: 8 total, 7 up, 8 in 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: failed to encode map e128 with expected crc 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: failed to encode map e128 with expected crc 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: failed to encode map e128 with expected crc 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: failed to encode map e128 with expected crc 2024-03-11T22:35:28.029 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:27 smithi073 ceph-mon[66481]: pgmap v224: 97 pgs: 5 peering, 9 stale+active+clean, 83 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 568 op/s 2024-03-11T22:35:28.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[59961]: 2024-03-11 22:35:28.107502514 +0000 UTC m=+0.677329210 container remove 15df440335ac83543edbe74c011f6a4af32fb3dcec9d185929709a077432dfee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , name=centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754) 2024-03-11T22:35:28.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 268816 9.95 MB/sec execute 524 sec latency 1078.063 ms 2024-03-11T22:35:28.772 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[60047]: 2024-03-11 22:35:28.395918888 +0000 UTC m=+0.025193231 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:35:28.772 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[60047]: 2024-03-11 22:35:28.516812847 +0000 UTC m=+0.146087182 container create 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T22:35:28.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:35:28.795 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: osdmap e129: 8 total, 7 up, 8 in 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:28.796 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:28 smithi073 ceph-mon[66481]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.128 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[60047]: 2024-03-11 22:35:28.792943659 +0000 UTC m=+0.422218008 container init 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:35:29.128 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[60047]: 2024-03-11 22:35:28.797243271 +0000 UTC m=+0.426517611 container start 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-11T22:35:29.128 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:28 smithi099 podman[60047]: 2024-03-11 22:35:28.903466974 +0000 UTC m=+0.532741310 container attach 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: osdmap e129: 8 total, 7 up, 8 in 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.129 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:28 smithi099 ceph-mon[53203]: failed to encode map e129 with expected crc 2024-03-11T22:35:29.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:29 smithi099 podman[60047]: 2024-03-11 22:35:29.122465124 +0000 UTC m=+0.751739451 container died 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T22:35:29.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 269168 9.95 MB/sec execute 525 sec latency 44.115 ms 2024-03-11T22:35:29.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:29 smithi073 ceph-mon[66481]: pgmap v226: 97 pgs: 6 active+undersized, 16 active+undersized+degraded, 5 peering, 1 stale+active+clean, 69 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 505 op/s; 134/1566 objects degraded (8.557%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:29.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:29 smithi099 ceph-mon[53203]: pgmap v226: 97 pgs: 6 active+undersized, 16 active+undersized+degraded, 5 peering, 1 stale+active+clean, 69 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 505 op/s; 134/1566 objects degraded (8.557%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:29.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:29 smithi099 podman[60072]: 2024-03-11 22:35:29.824461376 +0000 UTC m=+0.695739407 container remove 574564495e7fc5681864286554a4ad206e5467f218be03014401ae975e6c70dd (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T22:35:29.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:29 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-11T22:35:29.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:29 smithi099 systemd[1]: Stopped Ceph osd.5 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:35:29.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:29 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.5.service: Consumed 5min 39.247s CPU time. 2024-03-11T22:35:30.323 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 systemd[1]: Starting Ceph osd.5 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:35:30.323 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 podman[60172]: 2024-03-11 22:35:30.191079982 +0000 UTC m=+0.019573235 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:35:30.645 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 podman[60172]: 2024-03-11 22:35:30.317430403 +0000 UTC m=+0.145923636 container create fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T22:35:30.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 269632 9.96 MB/sec execute 526 sec latency 55.389 ms 2024-03-11T22:35:30.831 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:30 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 134/1566 objects degraded (8.557%), 16 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:30 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 134/1566 objects degraded (8.557%), 16 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:30.992 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 podman[60172]: 2024-03-11 22:35:30.637116241 +0000 UTC m=+0.465609465 container init fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:35:30.992 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 podman[60172]: 2024-03-11 22:35:30.641722401 +0000 UTC m=+0.470215625 container start fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T22:35:30.992 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:30 smithi099 podman[60172]: 2024-03-11 22:35:30.737271511 +0000 UTC m=+0.565764743 container attach fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:35:31.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T22:35:31.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 bash[60172]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate[60187]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-11T22:35:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 podman[60172]: 2024-03-11 22:35:31.136642265 +0000 UTC m=+0.965135481 container died fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T22:35:31.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 269955 9.96 MB/sec execute 527 sec latency 55.301 ms 2024-03-11T22:35:31.871 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:31 smithi073 ceph-mon[66481]: pgmap v227: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 5 peering, 66 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.2 MiB/s wr, 490 op/s; 149/1569 objects degraded (9.496%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:32.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:31 smithi099 ceph-mon[53203]: pgmap v227: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 5 peering, 66 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.2 MiB/s wr, 490 op/s; 149/1569 objects degraded (9.496%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:32.073 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:31 smithi099 podman[60319]: 2024-03-11 22:35:31.850430498 +0000 UTC m=+0.707508095 container remove fc268b46bf7d08598bd0494a1a1cdeb84cd92c15cb4ca88422036b0fd9595125 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T22:35:32.335 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 podman[60385]: 2024-03-11 22:35:31.967548317 +0000 UTC m=+0.018265078 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:35:32.335 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 podman[60385]: 2024-03-11 22:35:32.076451787 +0000 UTC m=+0.127168547 container create 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T22:35:32.742 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 podman[60385]: 2024-03-11 22:35:32.329096511 +0000 UTC m=+0.379813278 container init 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T22:35:32.742 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 podman[60385]: 2024-03-11 22:35:32.332513736 +0000 UTC m=+0.383230505 container start 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T22:35:32.742 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 ceph-osd[60404]: -- 172.21.15.99:0/1000505144 <== mon.1 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f0c77b1e0 con 0x564f0c74d000 2024-03-11T22:35:32.743 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 bash[60385]: 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c 2024-03-11T22:35:32.743 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 systemd[1]: Started Ceph osd.5 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:35:32.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 270251 9.96 MB/sec execute 528 sec latency 51.525 ms 2024-03-11T22:35:33.242 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:35:32.872+0000 7f2c4e16b7c0 -1 Falling back to public interface 2024-03-11T22:35:33.666 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:33 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:33.667 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:33 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:33.667 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:33 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:33.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 270514 9.96 MB/sec execute 529 sec latency 57.820 ms 2024-03-11T22:35:33.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:33 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:33.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:33 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:33.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:33 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:34.686 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:34 smithi073 ceph-mon[66481]: pgmap v228: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.2 MiB/s wr, 369 op/s; 180/1599 objects degraded (11.257%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:34.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:34 smithi099 ceph-mon[53203]: pgmap v228: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.2 MiB/s wr, 369 op/s; 180/1599 objects degraded (11.257%); 333 B/s, 0 objects/s recovering 2024-03-11T22:35:34.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 270881 9.96 MB/sec execute 530 sec latency 37.300 ms 2024-03-11T22:35:35.699 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:35 smithi073 ceph-mon[66481]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:35:35.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:35 smithi099 ceph-mon[53203]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:35:35.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 271168 9.95 MB/sec execute 531 sec latency 71.428 ms 2024-03-11T22:35:36.718 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:36 smithi073 ceph-mon[66481]: pgmap v229: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 416 op/s; 183/1668 objects degraded (10.971%); 300 B/s, 0 objects/s recovering 2024-03-11T22:35:36.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:36 smithi099 ceph-mon[53203]: pgmap v229: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 416 op/s; 183/1668 objects degraded (10.971%); 300 B/s, 0 objects/s recovering 2024-03-11T22:35:36.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 271398 9.94 MB/sec execute 532 sec latency 49.496 ms 2024-03-11T22:35:37.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 271714 9.93 MB/sec execute 533 sec latency 44.351 ms 2024-03-11T22:35:37.755 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:37 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 183/1668 objects degraded (10.971%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:37.755 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:37.755 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:37 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:37.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:37 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 183/1668 objects degraded (10.971%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:37.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:37.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:37 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:38.530 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:38 smithi073 ceph-mon[66481]: pgmap v230: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 380 op/s; 181/1662 objects degraded (10.890%); 267 B/s, 0 objects/s recovering 2024-03-11T22:35:38.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:38.531 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:38.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 272078 9.93 MB/sec execute 534 sec latency 54.327 ms 2024-03-11T22:35:38.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:38 smithi099 ceph-mon[53203]: pgmap v230: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.9 MiB/s wr, 380 op/s; 181/1662 objects degraded (10.890%); 267 B/s, 0 objects/s recovering 2024-03-11T22:35:38.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:38.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:39.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 272416 9.92 MB/sec execute 535 sec latency 92.140 ms 2024-03-11T22:35:39.809 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:39.809 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:39.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:39.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:40.710 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:35:40.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 272710 9.93 MB/sec execute 536 sec latency 49.738 ms 2024-03-11T22:35:40.852 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:40 smithi073 ceph-mon[66481]: pgmap v231: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 431 op/s; 181/1680 objects degraded (10.774%); 224 B/s, 0 objects/s recovering 2024-03-11T22:35:40.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:40 smithi099 ceph-mon[53203]: pgmap v231: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 431 op/s; 181/1680 objects degraded (10.774%); 224 B/s, 0 objects/s recovering 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (8m) 78s ago 17m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (4m) 78s ago 17m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (4m) 4s ago 16m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (7m) 78s ago 17m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (11m) 78s ago 11m 38.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (11m) 4s ago 11m 14.6M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (7m) 78s ago 18m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (6m) 4s ago 16m 451M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (5m) 78s ago 18m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (5m) 4s ago 16m 42.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (11m) 78s ago 11m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:35:41.605 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (11m) 4s ago 11m 86.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (7m) 78s ago 17m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (7m) 4s ago 16m 22.3M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (3m) 78s ago 15m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (3m) 78s ago 15m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (2m) 78s ago 14m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (88s) 78s ago 14m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (45s) 4s ago 14m 258M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (9s) 4s ago 13m 12.6M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (13m) 4s ago 13m 598M 2704M 17.2.0 e1d6a67b021e 218dd949f75d 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (12m) 4s ago 12m 651M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:35:41.606 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (6m) 78s ago 16m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:35:41.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 273138 9.93 MB/sec execute 537 sec latency 75.299 ms 2024-03-11T22:35:41.862 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:41 smithi073 ceph-mon[66481]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:41.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:41 smithi099 ceph-mon[53203]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:41.992 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:41 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:35:41.589+0000 7f2c4e16b7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:42.158 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 5 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 9 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:35:42.159 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:35:42.740 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "12/23 daemons upgraded", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:35:42.741 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: pgmap v232: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 410 op/s; 180/1686 objects degraded (10.676%) 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 180/1686 objects degraded (10.676%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1832836866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:35:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:42 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 273527 9.93 MB/sec execute 538 sec latency 45.868 ms 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: pgmap v232: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 410 op/s; 180/1686 objects degraded (10.676%) 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 180/1686 objects degraded (10.676%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1832836866' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:42.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:42 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:43.287 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 180/1686 objects degraded (10.676%), 22 pgs degraded 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: osd.5 (root=default,host=smithi099) is down 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 180/1686 objects degraded (10.676%), 22 pgs degraded 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-03-11T22:35:43.288 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-03-11T22:35:43.289 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-03-11T22:35:43.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:35:43.094+0000 7f2c4e16b7c0 -1 osd.5 127 log_to_monitors true 2024-03-11T22:35:43.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 273926 9.94 MB/sec execute 539 sec latency 40.702 ms 2024-03-11T22:35:43.917 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:35:43.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:43 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1976501284' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:35:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:35:43.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:43 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1976501284' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:35:44.492 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:35:44.213+0000 7f2c3b5bb700 -1 osd.5 127 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:35:44.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274162 9.93 MB/sec execute 540 sec latency 131.503 ms 2024-03-11T22:35:44.934 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:44.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:44.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-11T22:35:44.935 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: pgmap v233: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 MiB/s wr, 450 op/s; 175/1686 objects degraded (10.380%) 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: failed to encode map e130 with expected crc 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: osdmap e130: 8 total, 7 up, 8 in 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:44.937 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:44 smithi073 ceph-mon[66481]: from='osd.5 ' entity='osd.5' 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: pgmap v233: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 MiB/s wr, 450 op/s; 175/1686 objects degraded (10.380%) 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: failed to encode map e130 with expected crc 2024-03-11T22:35:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:44.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: osdmap e130: 8 total, 7 up, 8 in 2024-03-11T22:35:44.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:35:44.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:44 smithi099 ceph-mon[53203]: from='osd.5 ' entity='osd.5' 2024-03-11T22:35:45.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274464 9.93 MB/sec execute 541 sec latency 532.009 ms 2024-03-11T22:35:45.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:35:45.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: failed to encode map e131 with expected crc 2024-03-11T22:35:45.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422] boot 2024-03-11T22:35:45.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T22:35:45.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:35:45.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: failed to encode map e131 with expected crc 2024-03-11T22:35:45.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:45 smithi099 ceph-mon[53203]: pgmap v236: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.2 MiB/s wr, 495 op/s; 175/1659 objects degraded (10.549%) 2024-03-11T22:35:46.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:35:46.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: failed to encode map e131 with expected crc 2024-03-11T22:35:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: osd.5 [v2:172.21.15.99:6808/764135422,v1:172.21.15.99:6809/764135422] boot 2024-03-11T22:35:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T22:35:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:35:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: failed to encode map e131 with expected crc 2024-03-11T22:35:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:45 smithi073 ceph-mon[66481]: pgmap v236: 97 pgs: 9 active+undersized, 22 active+undersized+degraded, 66 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.2 MiB/s wr, 495 op/s; 175/1659 objects degraded (10.549%) 2024-03-11T22:35:46.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274485 9.92 MB/sec execute 542 sec latency 1469.252 ms 2024-03-11T22:35:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: failed to encode map e131 with expected crc 2024-03-11T22:35:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T22:35:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:46.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:46 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: failed to encode map e131 with expected crc 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:46 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274485 9.90 MB/sec execute 543 sec latency 2469.277 ms 2024-03-11T22:35:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: failed to encode map e132 with expected crc 2024-03-11T22:35:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: failed to encode map e133 with expected crc 2024-03-11T22:35:47.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T22:35:47.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: failed to encode map e133 with expected crc 2024-03-11T22:35:47.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:47 smithi099 ceph-mon[53203]: pgmap v239: 97 pgs: 1 remapped+peering, 3 peering, 6 active+undersized, 19 active+undersized+degraded, 68 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.7 MiB/s wr, 479 op/s; 153/1674 objects degraded (9.140%) 2024-03-11T22:35:48.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:48.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: failed to encode map e132 with expected crc 2024-03-11T22:35:48.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T22:35:48.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:47 smithi073 ceph-mon[66481]: pgmap v239: 97 pgs: 1 remapped+peering, 3 peering, 6 active+undersized, 19 active+undersized+degraded, 68 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.7 MiB/s wr, 479 op/s; 153/1674 objects degraded (9.140%) 2024-03-11T22:35:48.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274580 9.88 MB/sec execute 544 sec latency 3006.982 ms 2024-03-11T22:35:48.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:48 smithi073 ceph-mon[66481]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:48 smithi073 ceph-mon[66481]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:48 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 175/1659 objects degraded (10.549%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:48 smithi099 ceph-mon[53203]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:48 smithi099 ceph-mon[53203]: failed to encode map e133 with expected crc 2024-03-11T22:35:48.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:48 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 175/1659 objects degraded (10.549%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:49.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 274825 9.87 MB/sec execute 545 sec latency 426.916 ms 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.340 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:50 smithi073 ceph-mon[66481]: pgmap v241: 97 pgs: 1 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 4 peering, 1 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 79 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 458 B/s rd, 3.2 MiB/s wr, 361 op/s; 151/1617 objects degraded (9.338%); 938 KiB/s, 5 objects/s recovering 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: failed to encode map e134 with expected crc 2024-03-11T22:35:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:50 smithi099 ceph-mon[53203]: pgmap v241: 97 pgs: 1 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 4 peering, 1 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 79 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 458 B/s rd, 3.2 MiB/s wr, 361 op/s; 151/1617 objects degraded (9.338%); 938 KiB/s, 5 objects/s recovering 2024-03-11T22:35:50.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 275100 9.86 MB/sec execute 546 sec latency 931.657 ms 2024-03-11T22:35:51.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:51 smithi099 ceph-mon[53203]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:51 smithi099 ceph-mon[53203]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:51 smithi099 ceph-mon[53203]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T22:35:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:51 smithi099 ceph-mon[53203]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:51 smithi099 ceph-mon[53203]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:51 smithi073 ceph-mon[66481]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:51 smithi073 ceph-mon[66481]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:51 smithi073 ceph-mon[66481]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T22:35:51.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:51 smithi073 ceph-mon[66481]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.504 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:51 smithi073 ceph-mon[66481]: failed to encode map e135 with expected crc 2024-03-11T22:35:51.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 275328 9.85 MB/sec execute 547 sec latency 1000.916 ms 2024-03-11T22:35:52.259 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:52 smithi073 ceph-mon[66481]: failed to encode map e136 with expected crc 2024-03-11T22:35:52.259 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:52 smithi073 ceph-mon[66481]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T22:35:52.259 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:52 smithi073 ceph-mon[66481]: failed to encode map e136 with expected crc 2024-03-11T22:35:52.259 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:52 smithi073 ceph-mon[66481]: pgmap v244: 97 pgs: 1 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 87 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.9 MiB/s wr, 161 op/s; 110/1578 objects degraded (6.971%); 2.6 MiB/s, 19 objects/s recovering 2024-03-11T22:35:52.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:52 smithi099 ceph-mon[53203]: failed to encode map e136 with expected crc 2024-03-11T22:35:52.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:52 smithi099 ceph-mon[53203]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T22:35:52.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:52 smithi099 ceph-mon[53203]: failed to encode map e136 with expected crc 2024-03-11T22:35:52.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:52 smithi099 ceph-mon[53203]: pgmap v244: 97 pgs: 1 active+recovery_wait+degraded, 2 activating+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 87 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.9 MiB/s wr, 161 op/s; 110/1578 objects degraded (6.971%); 2.6 MiB/s, 19 objects/s recovering 2024-03-11T22:35:52.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 275661 9.84 MB/sec execute 548 sec latency 48.887 ms 2024-03-11T22:35:53.273 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:53 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 110/1578 objects degraded (6.971%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:53.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:53 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 110/1578 objects degraded (6.971%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T22:35:53.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 276047 9.85 MB/sec execute 549 sec latency 47.955 ms 2024-03-11T22:35:54.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 276450 9.85 MB/sec execute 550 sec latency 35.640 ms 2024-03-11T22:35:54.794 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:54 smithi073 ceph-mon[66481]: pgmap v245: 97 pgs: 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 88 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.0 MiB/s wr, 371 op/s; 86/1521 objects degraded (5.654%); 2.2 MiB/s, 18 objects/s recovering 2024-03-11T22:35:54.794 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:54.794 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:54 smithi099 ceph-mon[53203]: pgmap v245: 97 pgs: 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 88 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.0 MiB/s wr, 371 op/s; 86/1521 objects degraded (5.654%); 2.2 MiB/s, 18 objects/s recovering 2024-03-11T22:35:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:35:54.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:35:55.753 INFO:teuthology.orchestra.run.smithi073.stdout: 5 276794 9.85 MB/sec execute 551 sec latency 39.510 ms 2024-03-11T22:35:55.819 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:55 smithi073 ceph-mon[66481]: pgmap v246: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.9 MiB/s wr, 329 op/s; 2.2 MiB/s, 36 objects/s recovering 2024-03-11T22:35:55.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:55 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/1521 objects degraded (5.654%), 5 pgs degraded) 2024-03-11T22:35:55.820 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:55 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:35:56.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:55 smithi099 ceph-mon[53203]: pgmap v246: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.9 MiB/s wr, 329 op/s; 2.2 MiB/s, 36 objects/s recovering 2024-03-11T22:35:56.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:55 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 86/1521 objects degraded (5.654%), 5 pgs degraded) 2024-03-11T22:35:56.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:55 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:35:56.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 277177 9.87 MB/sec execute 552 sec latency 33.778 ms 2024-03-11T22:35:57.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 277473 9.87 MB/sec execute 553 sec latency 44.768 ms 2024-03-11T22:35:57.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:57 smithi073 ceph-mon[66481]: pgmap v247: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.0 MiB/s wr, 296 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-11T22:35:57.846 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:57 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:58.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:57 smithi099 ceph-mon[53203]: pgmap v247: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.0 MiB/s wr, 296 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-11T22:35:58.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:57 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:58.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 277776 9.87 MB/sec execute 554 sec latency 45.929 ms 2024-03-11T22:35:58.861 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:58 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:58.861 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:58 smithi073 ceph-mon[66481]: Upgrade: osd.6 is safe to restart 2024-03-11T22:35:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:58 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-11T22:35:58.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:58 smithi099 ceph-mon[53203]: Upgrade: osd.6 is safe to restart 2024-03-11T22:35:59.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 278030 9.86 MB/sec execute 555 sec latency 43.746 ms 2024-03-11T22:35:59.875 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:35:59 smithi073 ceph-mon[66481]: pgmap v248: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.1 MiB/s wr, 450 op/s; 312 KiB/s, 19 objects/s recovering 2024-03-11T22:36:00.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:35:59 smithi099 ceph-mon[53203]: pgmap v248: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.1 MiB/s wr, 450 op/s; 312 KiB/s, 19 objects/s recovering 2024-03-11T22:36:00.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 278271 9.85 MB/sec execute 556 sec latency 40.516 ms 2024-03-11T22:36:00.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:00 smithi073 ceph-mon[66481]: Upgrade: Updating osd.6 2024-03-11T22:36:00.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:00.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:36:00.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:00.920 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:00 smithi073 ceph-mon[66481]: Deploying daemon osd.6 on smithi099 2024-03-11T22:36:01.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:00 smithi099 ceph-mon[53203]: Upgrade: Updating osd.6 2024-03-11T22:36:01.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:01.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:36:01.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:01.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:00 smithi099 ceph-mon[53203]: Deploying daemon osd.6 on smithi099 2024-03-11T22:36:01.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 278569 9.84 MB/sec execute 557 sec latency 59.878 ms 2024-03-11T22:36:01.982 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:01 smithi073 ceph-mon[66481]: pgmap v249: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 432 op/s; 250 KiB/s, 16 objects/s recovering 2024-03-11T22:36:02.100 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:01 smithi099 ceph-mon[53203]: pgmap v249: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 432 op/s; 250 KiB/s, 16 objects/s recovering 2024-03-11T22:36:02.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 278889 9.83 MB/sec execute 558 sec latency 63.758 ms 2024-03-11T22:36:03.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 279169 9.82 MB/sec execute 559 sec latency 81.611 ms 2024-03-11T22:36:03.799 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:03 smithi073 ceph-mon[66481]: pgmap v250: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 471 op/s; 218 KiB/s, 14 objects/s recovering 2024-03-11T22:36:03.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:03 smithi099 ceph-mon[53203]: pgmap v250: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.6 MiB/s wr, 471 op/s; 218 KiB/s, 14 objects/s recovering 2024-03-11T22:36:03.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:03 smithi099 systemd[1]: Stopping Ceph osd.6 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:36:04.492 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:04 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[37346]: 2024-03-11T22:36:04.037+0000 7fc9157fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:36:04.492 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:04 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[37346]: 2024-03-11T22:36:04.037+0000 7fc9157fe700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-11T22:36:04.493 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:04 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[37346]: 2024-03-11T22:36:04.037+0000 7fc9157fe700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:36:04.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 279550 9.83 MB/sec execute 560 sec latency 43.529 ms 2024-03-11T22:36:04.832 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:04.832 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:04 smithi073 ceph-mon[66481]: osd.6 marked itself down and dead 2024-03-11T22:36:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:04.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:04 smithi099 ceph-mon[53203]: osd.6 marked itself down and dead 2024-03-11T22:36:05.742 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:05 smithi099 podman[62190]: 2024-03-11 22:36:05.451665716 +0000 UTC m=+1.536371267 container died 218dd949f75dc9f1a14068344ed9690e21a6ba0108475d767b750faa4e79a16e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8) 2024-03-11T22:36:05.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 279590 9.81 MB/sec execute 561 sec latency 959.113 ms 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: osdmap e137: 8 total, 7 up, 8 in 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.042 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:05 smithi073 ceph-mon[66481]: pgmap v252: 97 pgs: 10 stale+active+clean, 87 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: osdmap e137: 8 total, 7 up, 8 in 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: failed to encode map e137 with expected crc 2024-03-11T22:36:06.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:05 smithi099 ceph-mon[53203]: pgmap v252: 97 pgs: 10 stale+active+clean, 87 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 531 op/s 2024-03-11T22:36:06.420 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:06 smithi099 podman[62190]: 2024-03-11 22:36:06.146072891 +0000 UTC m=+2.230778451 container cleanup 218dd949f75dc9f1a14068344ed9690e21a6ba0108475d767b750faa4e79a16e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, ceph=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, CEPH_POINT_RELEASE=-17.2.0) 2024-03-11T22:36:06.421 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:06 smithi099 bash[62190]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6 2024-03-11T22:36:06.742 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:06 smithi099 podman[62206]: 2024-03-11 22:36:06.414524635 +0000 UTC m=+0.964446517 container remove 218dd949f75dc9f1a14068344ed9690e21a6ba0108475d767b750faa4e79a16e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., distribution-scope=public, architecture=x86_64) 2024-03-11T22:36:06.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 280022 9.81 MB/sec execute 562 sec latency 1040.168 ms 2024-03-11T22:36:07.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:06 smithi073 ceph-mon[66481]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:06 smithi073 ceph-mon[66481]: osdmap e138: 8 total, 7 up, 8 in 2024-03-11T22:36:07.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:06 smithi073 ceph-mon[66481]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:06 smithi073 ceph-mon[66481]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.052 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:06 smithi073 ceph-mon[66481]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.099 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:06 smithi099 ceph-mon[53203]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.099 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:06 smithi099 ceph-mon[53203]: osdmap e138: 8 total, 7 up, 8 in 2024-03-11T22:36:07.099 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:06 smithi099 ceph-mon[53203]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.099 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:06 smithi099 ceph-mon[53203]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.099 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:06 smithi099 ceph-mon[53203]: failed to encode map e138 with expected crc 2024-03-11T22:36:07.100 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:06 smithi099 podman[62293]: 2024-03-11 22:36:06.72535349 +0000 UTC m=+0.020378810 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:07.100 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:06 smithi099 podman[62293]: 2024-03-11 22:36:06.848691278 +0000 UTC m=+0.143716592 container create 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-11T22:36:07.100 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:07 smithi099 podman[62293]: 2024-03-11 22:36:07.093254973 +0000 UTC m=+0.388280284 container init 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T22:36:07.436 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:07 smithi099 podman[62293]: 2024-03-11 22:36:07.097230622 +0000 UTC m=+0.392255942 container start 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T22:36:07.437 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:07 smithi099 podman[62293]: 2024-03-11 22:36:07.185056584 +0000 UTC m=+0.480081902 container attach 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T22:36:07.437 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:07 smithi099 podman[62293]: 2024-03-11 22:36:07.430643924 +0000 UTC m=+0.725669238 container died 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T22:36:07.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 280371 9.82 MB/sec execute 563 sec latency 39.708 ms 2024-03-11T22:36:08.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:07 smithi073 ceph-mon[66481]: pgmap v254: 97 pgs: 5 active+undersized, 6 active+undersized+degraded, 4 peering, 5 stale+active+clean, 77 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 413 op/s; 36/1563 objects degraded (2.303%) 2024-03-11T22:36:08.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:07 smithi099 ceph-mon[53203]: pgmap v254: 97 pgs: 5 active+undersized, 6 active+undersized+degraded, 4 peering, 5 stale+active+clean, 77 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 413 op/s; 36/1563 objects degraded (2.303%) 2024-03-11T22:36:08.493 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62350]: 2024-03-11 22:36:08.106717942 +0000 UTC m=+0.670715115 container remove 8f5efc34c212bbaeeb363c11e77f0ef1be39da537445b43086035de4a310da22 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:36:08.493 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-11T22:36:08.494 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 systemd[1]: Stopped Ceph osd.6 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:36:08.494 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.6.service: Consumed 5min 25.288s CPU time. 2024-03-11T22:36:08.494 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 systemd[1]: Starting Ceph osd.6 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:36:08.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 280749 9.83 MB/sec execute 564 sec latency 36.261 ms 2024-03-11T22:36:08.815 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:08 smithi073 ceph-mon[66481]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:36:08.815 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:08 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 36/1563 objects degraded (2.303%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:08.816 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:08.816 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:08.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:08 smithi099 ceph-mon[53203]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:36:08.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:08 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 36/1563 objects degraded (2.303%), 6 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:08.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:08.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:08.818 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62449]: 2024-03-11 22:36:08.462589008 +0000 UTC m=+0.019642605 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:08.818 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62449]: 2024-03-11 22:36:08.582955161 +0000 UTC m=+0.140008743 container create 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T22:36:08.818 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62449]: 2024-03-11 22:36:08.811168284 +0000 UTC m=+0.368221879 container init 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T22:36:09.242 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62449]: 2024-03-11 22:36:08.816497633 +0000 UTC m=+0.373551217 container start 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:36:09.242 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:08 smithi099 podman[62449]: 2024-03-11 22:36:08.927721153 +0000 UTC m=+0.484774752 container attach 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:36:09.645 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 bash[62449]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-11T22:36:09.646 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-11T22:36:09.647 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate[62464]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-11T22:36:09.647 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:09 smithi099 podman[62449]: 2024-03-11 22:36:09.373228818 +0000 UTC m=+0.930282408 container died 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T22:36:09.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 281122 9.83 MB/sec execute 565 sec latency 46.132 ms 2024-03-11T22:36:09.888 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:09 smithi073 ceph-mon[66481]: pgmap v255: 97 pgs: 9 active+undersized, 17 active+undersized+degraded, 4 peering, 1 stale+active+clean, 66 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 558 op/s; 136/1620 objects degraded (8.395%) 2024-03-11T22:36:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:09 smithi099 ceph-mon[53203]: pgmap v255: 97 pgs: 9 active+undersized, 17 active+undersized+degraded, 4 peering, 1 stale+active+clean, 66 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 558 op/s; 136/1620 objects degraded (8.395%) 2024-03-11T22:36:10.377 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 podman[62596]: 2024-03-11 22:36:10.091062898 +0000 UTC m=+0.711901303 container remove 8d642c34a0c5edf1312218bf07ef5bba287a7168c2c16f60eed57bf0e03ca1b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T22:36:10.378 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 podman[62658]: 2024-03-11 22:36:10.23112947 +0000 UTC m=+0.020950247 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:10.378 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 podman[62658]: 2024-03-11 22:36:10.371716682 +0000 UTC m=+0.161537434 container create e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T22:36:10.629 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 podman[62658]: 2024-03-11 22:36:10.619911176 +0000 UTC m=+0.409731944 container init e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T22:36:10.629 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 podman[62658]: 2024-03-11 22:36:10.623082161 +0000 UTC m=+0.412902922 container start e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-11T22:36:10.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 281432 9.83 MB/sec execute 566 sec latency 47.545 ms 2024-03-11T22:36:10.992 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 bash[62658]: e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 2024-03-11T22:36:10.992 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:10 smithi099 systemd[1]: Started Ceph osd.6 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:36:11.492 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:36:11.175+0000 7f7c69cca7c0 -1 Falling back to public interface 2024-03-11T22:36:11.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 281677 9.82 MB/sec execute 567 sec latency 59.649 ms 2024-03-11T22:36:11.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:11.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:11.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:11.958 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:11 smithi073 ceph-mon[66481]: pgmap v256: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 452 op/s; 208/1662 objects degraded (12.515%) 2024-03-11T22:36:12.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:12.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:12.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:12.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:11 smithi099 ceph-mon[53203]: pgmap v256: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 452 op/s; 208/1662 objects degraded (12.515%) 2024-03-11T22:36:12.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 281962 9.81 MB/sec execute 568 sec latency 75.942 ms 2024-03-11T22:36:12.965 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:12 smithi073 ceph-mon[66481]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:36:13.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:12 smithi099 ceph-mon[53203]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T22:36:13.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 282264 9.80 MB/sec execute 569 sec latency 50.697 ms 2024-03-11T22:36:13.840 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:36:13.998 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:13 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 208/1662 objects degraded (12.515%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:13.998 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:13 smithi073 ceph-mon[66481]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.0 MiB/s wr, 351 op/s; 212/1677 objects degraded (12.642%) 2024-03-11T22:36:14.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:13 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 208/1662 objects degraded (12.515%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:14.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:13 smithi099 ceph-mon[53203]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.0 MiB/s wr, 351 op/s; 212/1677 objects degraded (12.642%) 2024-03-11T22:36:14.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 282541 9.80 MB/sec execute 570 sec latency 60.800 ms 2024-03-11T22:36:14.838 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:36:14.838 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (8m) 111s ago 18m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (5m) 111s ago 17m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (4m) 37s ago 17m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (8m) 111s ago 17m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (12m) 111s ago 12m 38.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (12m) 37s ago 12m 14.6M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (7m) 111s ago 19m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (6m) 37s ago 17m 451M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (6m) 111s ago 19m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (6m) 37s ago 17m 42.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (11m) 111s ago 11m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (11m) 37s ago 11m 86.2M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (8m) 111s ago 17m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (7m) 37s ago 17m 22.3M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (4m) 111s ago 16m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:36:14.839 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (3m) 111s ago 15m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (2m) 111s ago 15m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (2m) 111s ago 15m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (78s) 37s ago 14m 258M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (42s) 37s ago 14m 12.6M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 starting - - - 2704M 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (13m) 37s ago 13m 651M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:36:14.840 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (6m) 111s ago 16m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:36:15.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:14 smithi073 ceph-mon[66481]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:15.021 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:14 smithi073 ceph-mon[66481]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:14 smithi099 ceph-mon[53203]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:15.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:14 smithi099 ceph-mon[53203]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:15.430 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:36:15.430 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:36:15.430 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 6 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 10 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:36:15.431 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:36:15.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 282824 9.79 MB/sec execute 571 sec latency 54.813 ms 2024-03-11T22:36:15.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:15 smithi073 ceph-mon[66481]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:15.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:15 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2575280981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:15.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:15.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:15.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:15 smithi073 ceph-mon[66481]: pgmap v258: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 369 op/s; 210/1671 objects degraded (12.567%) 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:36:15.961 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "13/23 daemons upgraded", 2024-03-11T22:36:15.962 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:36:15.962 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:36:15.962 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:36:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:15 smithi099 ceph-mon[53203]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:15 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2575280981' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:15 smithi099 ceph-mon[53203]: pgmap v258: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 369 op/s; 210/1671 objects degraded (12.567%) 2024-03-11T22:36:16.586 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 210/1671 objects degraded (12.567%), 25 pgs degraded 2024-03-11T22:36:16.586 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T22:36:16.586 INFO:teuthology.orchestra.run.smithi073.stdout: osd.6 (root=default,host=smithi099) is down 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 210/1671 objects degraded (12.567%), 25 pgs degraded 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-03-11T22:36:16.587 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-03-11T22:36:16.588 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-03-11T22:36:16.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 283136 9.79 MB/sec execute 572 sec latency 107.839 ms 2024-03-11T22:36:17.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:17 smithi073 ceph-mon[66481]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:17.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:17 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2965937727' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:36:17.697 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:17 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:17.698 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:17 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:17.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 283465 9.78 MB/sec execute 573 sec latency 45.156 ms 2024-03-11T22:36:17.809 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:17 smithi099 ceph-mon[53203]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:17.809 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:17 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2965937727' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:36:17.809 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:17 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:17.809 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:17 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:18.731 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:18 smithi073 ceph-mon[66481]: pgmap v259: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 409 op/s; 195/1596 objects degraded (12.218%) 2024-03-11T22:36:18.731 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:18 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 210/1671 objects degraded (12.567%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:18.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 283803 9.79 MB/sec execute 574 sec latency 45.100 ms 2024-03-11T22:36:18.782 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:18 smithi099 ceph-mon[53203]: pgmap v259: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 409 op/s; 195/1596 objects degraded (12.218%) 2024-03-11T22:36:18.783 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:18 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 210/1671 objects degraded (12.567%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:18.783 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:18 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:36:18.718+0000 7f7c69cca7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-11T22:36:19.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284048 9.79 MB/sec execute 575 sec latency 59.763 ms 2024-03-11T22:36:19.999 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:19 smithi073 ceph-mon[66481]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 520 op/s; 188/1575 objects degraded (11.937%) 2024-03-11T22:36:20.052 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:19 smithi099 ceph-mon[53203]: pgmap v260: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 520 op/s; 188/1575 objects degraded (11.937%) 2024-03-11T22:36:20.384 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:36:20.044+0000 7f7c69cca7c0 -1 osd.6 136 log_to_monitors true 2024-03-11T22:36:20.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284408 9.80 MB/sec execute 576 sec latency 39.235 ms 2024-03-11T22:36:21.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:20 smithi073 ceph-mon[66481]: from='osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:36:21.063 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:20 smithi073 ceph-mon[66481]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:36:21.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:20 smithi099 ceph-mon[53203]: from='osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:36:21.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:20 smithi099 ceph-mon[53203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:36:21.742 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:36:21.263+0000 7f7c5711a700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:36:21.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284663 9.79 MB/sec execute 577 sec latency 192.208 ms 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: osdmap e139: 8 total, 7 up, 8 in 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: failed to encode map e139 with expected crc 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:21.836 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: from='osd.6 ' entity='osd.6' 2024-03-11T22:36:21.837 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:21 smithi073 ceph-mon[66481]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 482 op/s; 185/1608 objects degraded (11.505%) 2024-03-11T22:36:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: osdmap e139: 8 total, 7 up, 8 in 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: failed to encode map e139 with expected crc 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:22.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:22.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:22.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-11T22:36:22.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: from='osd.6 ' entity='osd.6' 2024-03-11T22:36:22.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:21 smithi099 ceph-mon[53203]: pgmap v262: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 482 op/s; 185/1608 objects degraded (11.505%) 2024-03-11T22:36:22.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284850 9.79 MB/sec execute 578 sec latency 492.065 ms 2024-03-11T22:36:23.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:22 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:36:23.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:22 smithi073 ceph-mon[66481]: osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116] boot 2024-03-11T22:36:23.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:22 smithi073 ceph-mon[66481]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T22:36:23.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:36:23.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:22 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:36:23.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:22 smithi099 ceph-mon[53203]: osd.6 [v2:172.21.15.99:6816/3394743116,v1:172.21.15.99:6817/3394743116] boot 2024-03-11T22:36:23.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:22 smithi099 ceph-mon[53203]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T22:36:23.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:36:23.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284850 9.77 MB/sec execute 579 sec latency 1492.094 ms 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 185/1608 objects degraded (11.505%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T22:36:24.081 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: pgmap v265: 97 pgs: 1 unknown, 5 peering, 8 active+undersized, 21 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 416 op/s; 156/1644 objects degraded (9.489%) 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:24.082 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e140 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 185/1608 objects degraded (11.505%), 25 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: failed to encode map e141 with expected crc 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: pgmap v265: 97 pgs: 1 unknown, 5 peering, 8 active+undersized, 21 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 416 op/s; 156/1644 objects degraded (9.489%) 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:24.754 INFO:teuthology.orchestra.run.smithi073.stdout: 5 284988 9.76 MB/sec execute 580 sec latency 1820.590 ms 2024-03-11T22:36:25.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:24 smithi073 ceph-mon[66481]: failed to encode map e142 with expected crc 2024-03-11T22:36:25.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:24 smithi073 ceph-mon[66481]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T22:36:25.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:24 smithi073 ceph-mon[66481]: failed to encode map e142 with expected crc 2024-03-11T22:36:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:24 smithi099 ceph-mon[53203]: failed to encode map e142 with expected crc 2024-03-11T22:36:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:24 smithi099 ceph-mon[53203]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T22:36:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:24 smithi099 ceph-mon[53203]: failed to encode map e142 with expected crc 2024-03-11T22:36:25.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 285036 9.74 MB/sec execute 581 sec latency 818.736 ms 2024-03-11T22:36:26.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:25 smithi073 ceph-mon[66481]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T22:36:26.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:25 smithi073 ceph-mon[66481]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:25 smithi073 ceph-mon[66481]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:25 smithi073 ceph-mon[66481]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.224 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:25 smithi073 ceph-mon[66481]: pgmap v268: 97 pgs: 2 remapped+peering, 1 unknown, 5 peering, 7 active+undersized, 1 active+undersized+remapped, 12 active+undersized+degraded, 69 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s wr, 278 op/s; 77/1656 objects degraded (4.650%); 37 KiB/s, 16 objects/s recovering 2024-03-11T22:36:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:25 smithi099 ceph-mon[53203]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T22:36:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:25 smithi099 ceph-mon[53203]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:25 smithi099 ceph-mon[53203]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:25 smithi099 ceph-mon[53203]: failed to encode map e143 with expected crc 2024-03-11T22:36:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:25 smithi099 ceph-mon[53203]: pgmap v268: 97 pgs: 2 remapped+peering, 1 unknown, 5 peering, 7 active+undersized, 1 active+undersized+remapped, 12 active+undersized+degraded, 69 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.9 MiB/s wr, 278 op/s; 77/1656 objects degraded (4.650%); 37 KiB/s, 16 objects/s recovering 2024-03-11T22:36:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 285220 9.73 MB/sec execute 582 sec latency 1054.409 ms 2024-03-11T22:36:27.001 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:26 smithi073 ceph-mon[66481]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T22:36:27.002 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:26 smithi073 ceph-mon[66481]: failed to encode map e144 with expected crc 2024-03-11T22:36:27.002 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:26 smithi073 ceph-mon[66481]: failed to encode map e144 with expected crc 2024-03-11T22:36:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:26 smithi099 ceph-mon[53203]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T22:36:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:26 smithi099 ceph-mon[53203]: failed to encode map e144 with expected crc 2024-03-11T22:36:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:26 smithi099 ceph-mon[53203]: failed to encode map e144 with expected crc 2024-03-11T22:36:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 285506 9.72 MB/sec execute 583 sec latency 56.434 ms 2024-03-11T22:36:28.019 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:27 smithi073 ceph-mon[66481]: pgmap v270: 97 pgs: 1 remapped+peering, 1 unknown, 7 peering, 1 active+undersized+remapped, 87 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.3 MiB/s wr, 198 op/s; 1.1 MiB/s, 57 keys/s, 29 objects/s recovering 2024-03-11T22:36:28.020 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:27 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 77/1656 objects degraded (4.650%), 12 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:27 smithi099 ceph-mon[53203]: pgmap v270: 97 pgs: 1 remapped+peering, 1 unknown, 7 peering, 1 active+undersized+remapped, 87 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.3 MiB/s wr, 198 op/s; 1.1 MiB/s, 57 keys/s, 29 objects/s recovering 2024-03-11T22:36:28.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:27 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 77/1656 objects degraded (4.650%), 12 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:28.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 285798 9.71 MB/sec execute 584 sec latency 44.483 ms 2024-03-11T22:36:29.049 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:28 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/1656 objects degraded (4.650%), 12 pgs degraded) 2024-03-11T22:36:29.050 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:28 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:36:29.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:28 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/1656 objects degraded (4.650%), 12 pgs degraded) 2024-03-11T22:36:29.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:28 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:36:29.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 286084 9.71 MB/sec execute 585 sec latency 87.321 ms 2024-03-11T22:36:30.070 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:29 smithi073 ceph-mon[66481]: pgmap v271: 97 pgs: 2 peering, 1 active+undersized+remapped, 94 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.4 MiB/s wr, 195 op/s; 1.3 MiB/s, 45 keys/s, 34 objects/s recovering 2024-03-11T22:36:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:29 smithi099 ceph-mon[53203]: pgmap v271: 97 pgs: 2 peering, 1 active+undersized+remapped, 94 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.4 MiB/s wr, 195 op/s; 1.3 MiB/s, 45 keys/s, 34 objects/s recovering 2024-03-11T22:36:30.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 286402 9.70 MB/sec execute 586 sec latency 38.093 ms 2024-03-11T22:36:31.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 286730 9.70 MB/sec execute 587 sec latency 67.663 ms 2024-03-11T22:36:31.828 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:31 smithi073 ceph-mon[66481]: pgmap v272: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.6 MiB/s wr, 312 op/s; 1008 KiB/s, 35 keys/s, 26 objects/s recovering 2024-03-11T22:36:32.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:31 smithi099 ceph-mon[53203]: pgmap v272: 97 pgs: 97 active+clean; 337 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.6 MiB/s wr, 312 op/s; 1008 KiB/s, 35 keys/s, 26 objects/s recovering 2024-03-11T22:36:32.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 287176 9.70 MB/sec execute 588 sec latency 35.829 ms 2024-03-11T22:36:33.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 287464 9.71 MB/sec execute 589 sec latency 52.139 ms 2024-03-11T22:36:33.870 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:33 smithi073 ceph-mon[66481]: pgmap v273: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.8 MiB/s wr, 385 op/s; 894 KiB/s, 31 keys/s, 18 objects/s recovering 2024-03-11T22:36:34.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:33 smithi099 ceph-mon[53203]: pgmap v273: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.8 MiB/s wr, 385 op/s; 894 KiB/s, 31 keys/s, 18 objects/s recovering 2024-03-11T22:36:34.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 287776 9.72 MB/sec execute 590 sec latency 45.691 ms 2024-03-11T22:36:35.755 INFO:teuthology.orchestra.run.smithi073.stdout: 5 288069 9.71 MB/sec execute 591 sec latency 61.382 ms 2024-03-11T22:36:35.927 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:35 smithi073 ceph-mon[66481]: pgmap v274: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.4 MiB/s wr, 387 op/s; 780 KiB/s, 27 keys/s, 16 objects/s recovering 2024-03-11T22:36:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:35 smithi099 ceph-mon[53203]: pgmap v274: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.4 MiB/s wr, 387 op/s; 780 KiB/s, 27 keys/s, 16 objects/s recovering 2024-03-11T22:36:36.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 288347 9.72 MB/sec execute 592 sec latency 42.770 ms 2024-03-11T22:36:36.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:36 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:36.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:36 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:36.947 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:36 smithi073 ceph-mon[66481]: Upgrade: osd.7 is safe to restart 2024-03-11T22:36:37.082 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:36 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:37.082 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:36 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-11T22:36:37.082 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:36 smithi099 ceph-mon[53203]: Upgrade: osd.7 is safe to restart 2024-03-11T22:36:37.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 288592 9.71 MB/sec execute 593 sec latency 46.237 ms 2024-03-11T22:36:37.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:37 smithi073 ceph-mon[66481]: pgmap v275: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 MiB/s wr, 353 op/s; 359 KiB/s, 9 objects/s recovering 2024-03-11T22:36:38.203 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:37 smithi099 ceph-mon[53203]: pgmap v275: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 MiB/s wr, 353 op/s; 359 KiB/s, 9 objects/s recovering 2024-03-11T22:36:38.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 288858 9.70 MB/sec execute 594 sec latency 48.811 ms 2024-03-11T22:36:39.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: Upgrade: Updating osd.7 2024-03-11T22:36:39.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: Deploying daemon osd.7 on smithi099 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: Upgrade: Updating osd.7 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: Deploying daemon osd.7 on smithi099 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:39.503 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:39.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 289163 9.69 MB/sec execute 595 sec latency 52.985 ms 2024-03-11T22:36:40.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:40 smithi099 ceph-mon[53203]: pgmap v276: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 MiB/s wr, 408 op/s; 228 KiB/s, 7 objects/s recovering 2024-03-11T22:36:40.506 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:40 smithi073 ceph-mon[66481]: pgmap v276: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 MiB/s wr, 408 op/s; 228 KiB/s, 7 objects/s recovering 2024-03-11T22:36:40.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 289491 9.69 MB/sec execute 596 sec latency 71.684 ms 2024-03-11T22:36:41.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 289882 9.68 MB/sec execute 597 sec latency 52.330 ms 2024-03-11T22:36:41.824 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:41 smithi073 ceph-mon[66481]: pgmap v277: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.5 MiB/s wr, 446 op/s; 7.9 KiB/s, 1 objects/s recovering 2024-03-11T22:36:42.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:41 smithi099 ceph-mon[53203]: pgmap v277: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.5 MiB/s wr, 446 op/s; 7.9 KiB/s, 1 objects/s recovering 2024-03-11T22:36:42.164 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:41 smithi099 systemd[1]: Stopping Ceph osd.7 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:36:42.492 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[40418]: 2024-03-11T22:36:42.156+0000 7fdee6b3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:36:42.492 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[40418]: 2024-03-11T22:36:42.156+0000 7fdee6b3a700 -1 osd.7 144 *** Got signal Terminated *** 2024-03-11T22:36:42.492 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[40418]: 2024-03-11T22:36:42.156+0000 7fdee6b3a700 -1 osd.7 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:36:42.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 290354 9.69 MB/sec execute 598 sec latency 36.865 ms 2024-03-11T22:36:42.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:42 smithi073 ceph-mon[66481]: osd.7 marked itself down and dead 2024-03-11T22:36:43.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:42 smithi099 ceph-mon[53203]: osd.7 marked itself down and dead 2024-03-11T22:36:43.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 290372 9.67 MB/sec execute 599 sec latency 971.150 ms 2024-03-11T22:36:43.779 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:43 smithi099 podman[64473]: 2024-03-11 22:36:43.503621079 +0000 UTC m=+1.482441879 container died a9b26ea36ae6738f4c33a302b0d19f31e78995a3c183e0d2600194fd59f3edb4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, io.k8s.display-name=CentOS Stream 8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, architecture=x86_64, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-11T22:36:44.046 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:43 smithi073 ceph-mon[66481]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:36:44.046 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:43 smithi073 ceph-mon[66481]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-11T22:36:44.046 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:43 smithi073 ceph-mon[66481]: osdmap e145: 8 total, 7 up, 8 in 2024-03-11T22:36:44.046 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:43 smithi073 ceph-mon[66481]: pgmap v279: 97 pgs: 8 peering, 11 stale+active+clean, 78 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 389 op/s 2024-03-11T22:36:44.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:43 smithi099 ceph-mon[53203]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T22:36:44.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:43 smithi099 ceph-mon[53203]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-11T22:36:44.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:43 smithi099 ceph-mon[53203]: osdmap e145: 8 total, 7 up, 8 in 2024-03-11T22:36:44.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:43 smithi099 ceph-mon[53203]: pgmap v279: 97 pgs: 8 peering, 11 stale+active+clean, 78 active+clean; 337 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 389 op/s 2024-03-11T22:36:44.243 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:43 smithi099 podman[64473]: 2024-03-11 22:36:43.872750025 +0000 UTC m=+1.851570834 container cleanup a9b26ea36ae6738f4c33a302b0d19f31e78995a3c183e0d2600194fd59f3edb4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, GIT_CLEAN=True, version=8, io.openshift.expose-services=, name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., architecture=x86_64, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T22:36:44.243 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:43 smithi099 bash[64473]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7 2024-03-11T22:36:44.741 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:44 smithi099 podman[64492]: 2024-03-11 22:36:44.324784633 +0000 UTC m=+0.822951323 container remove a9b26ea36ae6738f4c33a302b0d19f31e78995a3c183e0d2600194fd59f3edb4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64) 2024-03-11T22:36:44.758 INFO:teuthology.orchestra.run.smithi073.stdout: 5 cleanup 600 sec 2024-03-11T22:36:44.816 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:44 smithi073 ceph-mon[66481]: osdmap e146: 8 total, 7 up, 8 in 2024-03-11T22:36:44.961 INFO:teuthology.orchestra.run.smithi073.stdout: 0 cleanup 600 sec 2024-03-11T22:36:44.961 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:36:44.961 INFO:teuthology.orchestra.run.smithi073.stdout: Operation Count AvgLat MaxLat 2024-03-11T22:36:44.961 INFO:teuthology.orchestra.run.smithi073.stdout: ---------------------------------------- 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: NTCreateX 186586 4.585 2765.344 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Close 137143 0.131 8.696 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Rename 7908 10.465 1153.646 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Unlink 37643 7.433 2811.293 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Qpathinfo 169402 1.136 1811.873 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Qfileinfo 29553 0.008 0.852 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Qfsinfo 30966 0.631 28.576 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Sfileinfo 15273 12.731 88.677 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: Find 65406 0.926 19.036 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: WriteX 92264 13.984 3006.977 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: ReadX 292536 0.010 8.320 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: LockX 608 0.357 1.643 2024-03-11T22:36:44.962 INFO:teuthology.orchestra.run.smithi073.stdout: UnlockX 608 0.313 1.720 2024-03-11T22:36:44.963 INFO:teuthology.orchestra.run.smithi073.stdout: Flush 13110 0.002 0.031 2024-03-11T22:36:44.963 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:36:44.963 INFO:teuthology.orchestra.run.smithi073.stdout:Throughput 9.67073 MB/sec 5 clients 5 procs max_latency=3006.982 ms 2024-03-11T22:36:44.964 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-03-11T22:36:44.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:44 smithi099 ceph-mon[53203]: osdmap e146: 8 total, 7 up, 8 in 2024-03-11T22:36:44.992 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:44 smithi099 podman[64582]: 2024-03-11 22:36:44.639164237 +0000 UTC m=+0.021771539 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:44.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:44 smithi099 podman[64582]: 2024-03-11 22:36:44.781076977 +0000 UTC m=+0.163684270 container create 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T22:36:44.999 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-03-11T22:36:45.134 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-03-11T22:36:45.160 INFO:teuthology.orchestra.run.smithi073.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-11T22:36:45.160 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:36:45.160 INFO:teuthology.orchestra.run.smithi073.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-03-11T22:36:45.161 INFO:teuthology.orchestra.run.smithi073.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-11T22:36:45.261 INFO:teuthology.orchestra.run.smithi073.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-11T22:36:45.261 INFO:teuthology.orchestra.run.smithi073.stdout:releasing clients 2024-03-11T22:36:45.314 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:44 smithi099 podman[64582]: 2024-03-11 22:36:44.996167688 +0000 UTC m=+0.378774986 container init 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:36:45.314 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:44 smithi099 podman[64582]: 2024-03-11 22:36:44.999385718 +0000 UTC m=+0.381993012 container start 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:36:45.314 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 podman[64582]: 2024-03-11 22:36:45.095988545 +0000 UTC m=+0.478595852 container attach 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:36:45.742 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 podman[64582]: 2024-03-11 22:36:45.308502065 +0000 UTC m=+0.691109367 container died 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T22:36:45.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:45 smithi073 ceph-mon[66481]: pgmap v281: 97 pgs: 1 active+undersized+degraded, 24 peering, 7 stale+active+clean, 65 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 481 op/s; 6/1584 objects degraded (0.379%) 2024-03-11T22:36:46.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:45 smithi099 ceph-mon[53203]: pgmap v281: 97 pgs: 1 active+undersized+degraded, 24 peering, 7 stale+active+clean, 65 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 481 op/s; 6/1584 objects degraded (0.379%) 2024-03-11T22:36:46.148 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 podman[64609]: 2024-03-11 22:36:45.892941939 +0000 UTC m=+0.579041160 container remove 633ac0af27b49a6cc4339479f9c66aefc59564123cd0258726a293599d2a9670 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:36:46.148 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-11T22:36:46.148 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 systemd[1]: Stopped Ceph osd.7 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:36:46.148 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:45 smithi099 systemd[1]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.7.service: Consumed 8min 3.167s CPU time. 2024-03-11T22:36:46.409 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 systemd[1]: Starting Ceph osd.7 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:36:46.409 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 podman[64705]: 2024-03-11 22:36:46.287683976 +0000 UTC m=+0.022408696 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:46.671 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 podman[64705]: 2024-03-11 22:36:46.403096686 +0000 UTC m=+0.137821380 container create ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T22:36:46.956 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:46 smithi073 ceph-mon[66481]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:36:46.956 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:46 smithi073 ceph-mon[66481]: Health check failed: Degraded data redundancy: 6/1584 objects degraded (0.379%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:36:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:46 smithi099 ceph-mon[53203]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:36:46.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:46 smithi099 ceph-mon[53203]: Health check failed: Degraded data redundancy: 6/1584 objects degraded (0.379%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:36:46.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 podman[64705]: 2024-03-11 22:36:46.66504407 +0000 UTC m=+0.399768791 container init ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:36:46.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 podman[64705]: 2024-03-11 22:36:46.67482715 +0000 UTC m=+0.409551858 container start ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T22:36:46.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:46 smithi099 podman[64705]: 2024-03-11 22:36:46.773219814 +0000 UTC m=+0.507944518 container attach ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T22:36:47.007 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:36:47.261 INFO:teuthology.orchestra.run.smithi073.stdout: 5 341 22.37 MB/sec execute 1 sec latency 127.216 ms 2024-03-11T22:36:47.492 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 bash[64705]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-11T22:36:47.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate[64719]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-11T22:36:47.494 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 podman[64705]: 2024-03-11 22:36:47.261097144 +0000 UTC m=+0.995821831 container died ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (9m) 2m ago 18m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (5m) 2m ago 18m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (5m) 32s ago 17m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (9m) 2m ago 18m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (12m) 2m ago 12m 38.2M - 17.2.0 e1d6a67b021e 79aaaca93a38 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (12m) 32s ago 12m 14.7M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (8m) 2m ago 19m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:36:47.830 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (7m) 32s ago 17m 451M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (7m) 2m ago 19m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (6m) 32s ago 17m 43.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (12m) 2m ago 12m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (12m) 32s ago 12m 86.3M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (8m) 2m ago 18m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (8m) 32s ago 17m 18.4M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (4m) 2m ago 16m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (4m) 2m ago 16m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (3m) 2m ago 16m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (2m) 2m ago 15m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (111s) 32s ago 15m 297M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (75s) 32s ago 14m 258M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:36:47.831 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (37s) 32s ago 14m 12.4M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:36:47.832 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (14m) 32s ago 14m 698M 2704M 17.2.0 e1d6a67b021e a9b26ea36ae6 2024-03-11T22:36:47.832 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (7m) 2m ago 17m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:36:48.019 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:47 smithi073 ceph-mon[66481]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:48.020 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:47 smithi073 ceph-mon[66481]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:48.020 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:47 smithi073 ceph-mon[66481]: pgmap v282: 97 pgs: 18 active+undersized+degraded, 24 peering, 55 active+clean; 351 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.7 MiB/s wr, 455 op/s; 136/1668 objects degraded (8.153%) 2024-03-11T22:36:48.140 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:47 smithi099 podman[64837]: 2024-03-11 22:36:47.878936208 +0000 UTC m=+0.610980617 container remove ecad98ea342861526345c23a0abd9e68e4a101c30e9d34901399b27dab5c9da0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T22:36:48.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-mon[53203]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:48.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-mon[53203]: from='client.34442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:48.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:47 smithi099 ceph-mon[53203]: pgmap v282: 97 pgs: 18 active+undersized+degraded, 24 peering, 55 active+clean; 351 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.7 MiB/s wr, 455 op/s; 136/1668 objects degraded (8.153%) 2024-03-11T22:36:48.261 INFO:teuthology.orchestra.run.smithi073.stdout: 5 460 20.71 MB/sec execute 2 sec latency 55.597 ms 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:48.367 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 7 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 11 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:36:48.368 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:36:48.406 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 podman[64920]: 2024-03-11 22:36:48.039622586 +0000 UTC m=+0.025026821 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:36:48.406 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 podman[64920]: 2024-03-11 22:36:48.163520349 +0000 UTC m=+0.148924576 container create f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:36:48.407 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 podman[64920]: 2024-03-11 22:36:48.37483461 +0000 UTC m=+0.360238827 container init f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T22:36:48.407 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 podman[64920]: 2024-03-11 22:36:48.377943472 +0000 UTC m=+0.363347698 container start f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T22:36:48.742 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-osd[64938]: -- 172.21.15.99:0/1010197615 <== mon.1 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55af99e491e0 con 0x55af99e1ac00 2024-03-11T22:36:48.742 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 bash[64920]: f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa 2024-03-11T22:36:48.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 systemd[1]: Started Ceph osd.7 for f2f121a8-dff4-11ee-95c8-87774f69a715. 2024-03-11T22:36:48.809 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:36:48.809 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:36:48.809 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:36:48.809 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:36:48.809 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "13/23 daemons upgraded", 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading osd daemons", 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:36:48.810 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:36:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:48 smithi073 ceph-mon[66481]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:48 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1763829091' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:49.075 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:49.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-mon[53203]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:49.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1763829091' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:49.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:49.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:49.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:49.243 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:48 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:36:48.917+0000 7f79e87217c0 -1 Falling back to public interface 2024-03-11T22:36:49.261 INFO:teuthology.orchestra.run.smithi073.stdout: 5 620 20.80 MB/sec execute 3 sec latency 177.859 ms 2024-03-11T22:36:49.492 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg inactive, 2 pgs peering; Degraded data redundancy: 136/1668 objects degraded (8.153%), 18 pgs degraded 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: osd.7 (root=default,host=smithi099) is down 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 1.0 is stuck peering for 6m, current state peering, last acting [0,6] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1c is stuck peering for 63s, current state peering, last acting [5,2] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 136/1668 objects degraded (8.153%), 18 pgs degraded 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-11T22:36:49.493 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-11T22:36:49.494 INFO:teuthology.orchestra.run.smithi073.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-03-11T22:36:50.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:49 smithi073 ceph-mon[66481]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:50.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:49 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1424774006' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:36:50.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:49 smithi073 ceph-mon[66481]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 346 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 489 op/s; 244/1521 objects degraded (16.042%) 2024-03-11T22:36:50.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:49 smithi099 ceph-mon[53203]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:36:50.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:49 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1424774006' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:36:50.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:49 smithi099 ceph-mon[53203]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 346 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s wr, 489 op/s; 244/1521 objects degraded (16.042%) 2024-03-11T22:36:50.261 INFO:teuthology.orchestra.run.smithi073.stdout: 5 869 17.19 MB/sec execute 4 sec latency 51.216 ms 2024-03-11T22:36:50.899 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:50 smithi073 ceph-mon[66481]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-11T22:36:51.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:50 smithi099 ceph-mon[53203]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-11T22:36:51.261 INFO:teuthology.orchestra.run.smithi073.stdout: 5 cleanup 5 sec 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: 0 cleanup 5 sec 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: Operation Count AvgLat MaxLat 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: ---------------------------------------- 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: NTCreateX 679 9.786 68.300 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: Close 636 10.043 177.856 2024-03-11T22:36:51.513 INFO:teuthology.orchestra.run.smithi073.stdout: Rename 34 19.142 51.209 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Unlink 61 3.372 22.974 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Qpathinfo 306 1.628 63.026 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Qfileinfo 333 19.660 145.014 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Qfsinfo 132 0.367 1.666 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Sfileinfo 121 16.207 64.632 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Find 164 0.708 2.588 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: WriteX 1633 0.026 0.088 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: ReadX 582 0.021 0.302 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: Flush 120 16.189 27.930 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:36:51.514 INFO:teuthology.orchestra.run.smithi073.stdout:Throughput 17.1948 MB/sec 5 clients 5 procs max_latency=177.859 ms 2024-03-11T22:36:51.515 DEBUG:teuthology.parallel:result is None 2024-03-11T22:36:52.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:51 smithi073 ceph-mon[66481]: pgmap v284: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 12 MiB/s wr, 599 op/s; 243/1467 objects degraded (16.564%) 2024-03-11T22:36:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:51 smithi099 ceph-mon[53203]: pgmap v284: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 12 MiB/s wr, 599 op/s; 243/1467 objects degraded (16.564%) 2024-03-11T22:36:53.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:52 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 243/1467 objects degraded (16.564%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:53.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:52 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 243/1467 objects degraded (16.564%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:54.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:54.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:54 smithi073 ceph-mon[66481]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 315 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 12 MiB/s wr, 470 op/s; 195/1305 objects degraded (14.943%) 2024-03-11T22:36:54.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:54.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:36:54.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:54 smithi099 ceph-mon[53203]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 315 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 12 MiB/s wr, 470 op/s; 195/1305 objects degraded (14.943%) 2024-03-11T22:36:55.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:55.971 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:55.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:55.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:56.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:56 smithi073 ceph-mon[66481]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 308 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 11 MiB/s wr, 401 op/s; 180/1179 objects degraded (15.267%) 2024-03-11T22:36:56.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:56 smithi099 ceph-mon[53203]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 308 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 11 MiB/s wr, 401 op/s; 180/1179 objects degraded (15.267%) 2024-03-11T22:36:57.742 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:57 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:36:57.364+0000 7f79e87217c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-11T22:36:58.090 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:57 smithi099 ceph-mon[53203]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 276 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 336 op/s; 128/822 objects degraded (15.572%); 614 B/s, 0 objects/s recovering 2024-03-11T22:36:58.091 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:57 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 128/822 objects degraded (15.572%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:58.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:57 smithi073 ceph-mon[66481]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 276 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 336 op/s; 128/822 objects degraded (15.572%); 614 B/s, 0 objects/s recovering 2024-03-11T22:36:58.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:57 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 128/822 objects degraded (15.572%), 35 pgs degraded (PG_DEGRADED) 2024-03-11T22:36:59.242 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:58 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:36:58.898+0000 7f79e87217c0 -1 osd.7 144 log_to_monitors true 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all osd 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-11T22:36:59.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: Upgrade: Setting require_osd_release to 19 squid 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:36:59.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-mon[53203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all osd 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-11T22:36:59.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: Upgrade: Setting require_osd_release to 19 squid 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:36:59.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:36:59 smithi073 ceph-mon[66481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:37:00.149 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:36:59 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:36:59.818+0000 7f79d5b71700 -1 osd.7 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:37:00.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-11T22:37:00.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: osdmap e147: 8 total, 7 up, 8 in 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi073.lmopce"]}]: dispatch 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: Upgrade: It appears safe to stop mds.foofs.smithi073.lmopce 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: pgmap v289: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 228 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 259 op/s; 80/456 objects degraded (17.544%) 2024-03-11T22:37:00.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:00 smithi073 ceph-mon[66481]: from='osd.7 ' entity='osd.7' 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: osdmap e147: 8 total, 7 up, 8 in 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi073.lmopce"]}]: dispatch 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: Upgrade: It appears safe to stop mds.foofs.smithi073.lmopce 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: pgmap v289: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 228 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 259 op/s; 80/456 objects degraded (17.544%) 2024-03-11T22:37:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:00 smithi099 ceph-mon[53203]: from='osd.7 ' entity='osd.7' 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606] boot 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:37:01.687 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:01 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:01.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T22:37:01.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: osd.7 [v2:172.21.15.99:6824/157822606,v1:172.21.15.99:6825/157822606] boot 2024-03-11T22:37:01.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T22:37:01.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:37:01.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:01.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi073.lmopce", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:37:01.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:01 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:02.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:02 smithi073 ceph-mon[66481]: Upgrade: Updating mds.foofs.smithi073.lmopce 2024-03-11T22:37:02.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:02 smithi073 ceph-mon[66481]: Deploying daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:37:02.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:02 smithi073 ceph-mon[66481]: pgmap v291: 97 pgs: 4 peering, 9 active+undersized, 29 active+undersized+degraded, 55 active+clean; 228 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 84 op/s; 68/453 objects degraded (15.011%) 2024-03-11T22:37:02.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:02 smithi073 ceph-mon[66481]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T22:37:02.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:02 smithi099 ceph-mon[53203]: Upgrade: Updating mds.foofs.smithi073.lmopce 2024-03-11T22:37:02.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:02 smithi099 ceph-mon[53203]: Deploying daemon mds.foofs.smithi073.lmopce on smithi073 2024-03-11T22:37:02.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:02 smithi099 ceph-mon[53203]: pgmap v291: 97 pgs: 4 peering, 9 active+undersized, 29 active+undersized+degraded, 55 active+clean; 228 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 84 op/s; 68/453 objects degraded (15.011%) 2024-03-11T22:37:02.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:02 smithi099 ceph-mon[53203]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T22:37:03.822 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[66477]: 2024-03-11T22:37:03.530+0000 7f6e2205e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:04.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 68/453 objects degraded (15.011%), 29 pgs degraded (PG_DEGRADED) 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Standby daemon mds.foofs.smithi099.kagazk assigned to filesystem foofs as rank 0 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:replay} 2024-03-11T22:37:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:03 smithi073 ceph-mon[66481]: pgmap v295: 97 pgs: 1 activating+degraded, 2 activating, 4 peering, 7 active+undersized, 21 active+undersized+degraded, 62 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 70 KiB/s wr, 4 op/s; 96/432 objects degraded (22.222%); 0 B/s, 12 objects/s recovering 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 68/453 objects degraded (15.011%), 29 pgs degraded (PG_DEGRADED) 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Standby daemon mds.foofs.smithi099.kagazk assigned to filesystem foofs as rank 0 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:replay} 2024-03-11T22:37:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:03 smithi099 ceph-mon[53203]: pgmap v295: 97 pgs: 1 activating+degraded, 2 activating, 4 peering, 7 active+undersized, 21 active+undersized+degraded, 62 active+clean; 223 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 70 KiB/s wr, 4 op/s; 96/432 objects degraded (22.222%); 0 B/s, 12 objects/s recovering 2024-03-11T22:37:04.969 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:04 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:04.563+0000 7f962dd8c700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 19 2024-03-11T22:37:04.970 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:04 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:04.563+0000 7f9630d92700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 16 2024-03-11T22:37:04.970 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:04 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:04.563+0000 7f962d58b700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 19 2024-03-11T22:37:04.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:37:04 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:37:04.560+0000 7f2c29d98700 -1 bdev(0x564f0d548000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-03-11T22:37:05.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:05 smithi099 ceph-mon[53203]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T22:37:05.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:05.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:05 smithi073 ceph-mon[66481]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T22:37:05.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:06.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:06 smithi073 ceph-mon[66481]: reconnect by client.24349 172.21.15.99:0/589926945 after 0.000999978 2024-03-11T22:37:06.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:06 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:reconnect 2024-03-11T22:37:06.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:06 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:reconnect} 2024-03-11T22:37:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:06 smithi073 ceph-mon[66481]: reconnect by client.14540 172.21.15.73:0/239032306 after 0.00799983 2024-03-11T22:37:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:06 smithi073 ceph-mon[66481]: pgmap v297: 97 pgs: 1 activating+degraded, 2 activating, 5 peering, 3 active+undersized+degraded, 86 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 2 op/s; 60/426 objects degraded (14.085%); 286 KiB/s, 70 objects/s recovering 2024-03-11T22:37:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:06 smithi099 ceph-mon[53203]: reconnect by client.24349 172.21.15.99:0/589926945 after 0.000999978 2024-03-11T22:37:06.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:06 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:reconnect 2024-03-11T22:37:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:06 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:reconnect} 2024-03-11T22:37:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:06 smithi099 ceph-mon[53203]: reconnect by client.14540 172.21.15.73:0/239032306 after 0.00799983 2024-03-11T22:37:06.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:06 smithi099 ceph-mon[53203]: pgmap v297: 97 pgs: 1 activating+degraded, 2 activating, 5 peering, 3 active+undersized+degraded, 86 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 2 op/s; 60/426 objects degraded (14.085%); 286 KiB/s, 70 objects/s recovering 2024-03-11T22:37:07.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:07 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:rejoin 2024-03-11T22:37:07.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:07 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:rejoin} 2024-03-11T22:37:07.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:07 smithi073 ceph-mon[66481]: daemon mds.foofs.smithi099.kagazk is now active in filesystem foofs as rank 0 2024-03-11T22:37:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:07 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:rejoin 2024-03-11T22:37:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:07 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi099.kagazk=up:rejoin} 2024-03-11T22:37:07.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:07 smithi099 ceph-mon[53203]: daemon mds.foofs.smithi099.kagazk is now active in filesystem foofs as rank 0 2024-03-11T22:37:08.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:08 smithi073 ceph-mon[66481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T22:37:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:08 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:active 2024-03-11T22:37:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:08 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi099.kagazk=up:active} 2024-03-11T22:37:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:08 smithi073 ceph-mon[66481]: pgmap v298: 97 pgs: 1 peering, 96 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 6.9 KiB/s wr, 10 op/s; 948 KiB/s, 56 objects/s recovering 2024-03-11T22:37:08.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:08 smithi073 ceph-mon[66481]: Health check update: Degraded data redundancy: 60/426 objects degraded (14.085%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T22:37:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:08 smithi099 ceph-mon[53203]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T22:37:08.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:08 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.99:6832/485611820,v1:172.21.15.99:6833/485611820] up:active 2024-03-11T22:37:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:08 smithi099 ceph-mon[53203]: fsmap foofs:1 {0=foofs.smithi099.kagazk=up:active} 2024-03-11T22:37:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:08 smithi099 ceph-mon[53203]: pgmap v298: 97 pgs: 1 peering, 96 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 6.9 KiB/s wr, 10 op/s; 948 KiB/s, 56 objects/s recovering 2024-03-11T22:37:08.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:08 smithi099 ceph-mon[53203]: Health check update: Degraded data redundancy: 60/426 objects degraded (14.085%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T22:37:09.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:09 smithi073 ceph-mon[66481]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/426 objects degraded (14.085%), 4 pgs degraded) 2024-03-11T22:37:09.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:09.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:09 smithi099 ceph-mon[53203]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/426 objects degraded (14.085%), 4 pgs degraded) 2024-03-11T22:37:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:09.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:10.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:10 smithi073 ceph-mon[66481]: pgmap v299: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 5.9 KiB/s wr, 17 op/s; 811 KiB/s, 47 objects/s recovering 2024-03-11T22:37:10.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:10 smithi099 ceph-mon[53203]: pgmap v299: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 5.9 KiB/s wr, 17 op/s; 811 KiB/s, 47 objects/s recovering 2024-03-11T22:37:12.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:11 smithi073 ceph-mon[66481]: pgmap v300: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 5.3 KiB/s wr, 15 op/s; 684 KiB/s, 34 objects/s recovering 2024-03-11T22:37:12.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:11 smithi099 ceph-mon[53203]: pgmap v300: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 5.3 KiB/s wr, 15 op/s; 684 KiB/s, 34 objects/s recovering 2024-03-11T22:37:14.199 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:13 smithi073 ceph-mon[66481]: pgmap v301: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 4.3 KiB/s wr, 12 op/s; 559 KiB/s, 28 objects/s recovering 2024-03-11T22:37:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:13 smithi099 ceph-mon[53203]: pgmap v301: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 4.3 KiB/s wr, 12 op/s; 559 KiB/s, 28 objects/s recovering 2024-03-11T22:37:15.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:15 smithi073 ceph-mon[66481]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T22:37:15.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:15 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:37:15.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:15 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:boot 2024-03-11T22:37:15.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:15 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi099.kagazk=up:active} 1 up:standby 2024-03-11T22:37:15.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:15 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:15 smithi099 ceph-mon[53203]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T22:37:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:15 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:37:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:15 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:boot 2024-03-11T22:37:15.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:15 smithi099 ceph-mon[53203]: fsmap foofs:1 {0=foofs.smithi099.kagazk=up:active} 1 up:standby 2024-03-11T22:37:15.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:15 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:16.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:16.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:16.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:16.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:16 smithi073 ceph-mon[66481]: pgmap v302: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.8 KiB/s wr, 11 op/s; 398 KiB/s, 4 objects/s recovering 2024-03-11T22:37:16.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:16.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:16.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:16.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:16 smithi099 ceph-mon[53203]: pgmap v302: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.8 KiB/s wr, 11 op/s; 398 KiB/s, 4 objects/s recovering 2024-03-11T22:37:17.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:17 smithi073 ceph-mon[66481]: pgmap v303: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 11 KiB/s wr, 13 op/s; 371 KiB/s, 4 objects/s recovering 2024-03-11T22:37:18.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:17 smithi099 ceph-mon[53203]: pgmap v303: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 11 KiB/s wr, 13 op/s; 371 KiB/s, 4 objects/s recovering 2024-03-11T22:37:19.878 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:37:20.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:19 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:20.186 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:19 smithi073 ceph-mon[66481]: pgmap v304: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-11T22:37:20.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:19 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:20.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:19 smithi099 ceph-mon[53203]: pgmap v304: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (10m) 2m ago 19m 22.5M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (6m) 2m ago 19m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (6m) 27s ago 18m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (9m) 2m ago 18m 44.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 starting - - - - 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (13m) 27s ago 13m 14.8M - 17.2.0 e1d6a67b021e d1f18e59ebbe 2024-03-11T22:37:20.551 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (8m) 2m ago 20m 503M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (7m) 27s ago 18m 451M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (7m) 2m ago 20m 49.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (7m) 27s ago 18m 45.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (12m) 2m ago 12m 281M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (12m) 27s ago 12m 86.3M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (9m) 2m ago 18m 9831k - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (9m) 27s ago 18m 18.2M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (5m) 2m ago 17m 337M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (4m) 2m ago 16m 288M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (3m) 2m ago 16m 233M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (3m) 2m ago 16m 12.5M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (2m) 27s ago 15m 337M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (108s) 27s ago 15m 308M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (70s) 27s ago 14m 290M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (32s) 27s ago 14m 12.4M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:37:20.552 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (8m) 2m ago 17m 63.3M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:37:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 1 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:37:20.959 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-11T22:37:20.960 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 13 2024-03-11T22:37:20.960 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:37:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:37:21.309 INFO:teuthology.orchestra.run.smithi073.stdout: "osd", 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "15/23 daemons upgraded", 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading mds daemons", 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:37:21.310 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:37:21.711 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:37:21.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:21.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:21.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:21 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/504281756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:21 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/504281756' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:22.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:22 smithi073 ceph-mon[66481]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:22.788 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:22 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2129456953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:37:22.789 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:22 smithi073 ceph-mon[66481]: pgmap v305: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 4 op/s 2024-03-11T22:37:22.789 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:22.789 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:22 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:22.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:22 smithi099 ceph-mon[53203]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:22.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:22 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2129456953' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:37:22.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:22 smithi099 ceph-mon[53203]: pgmap v305: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 4 op/s 2024-03-11T22:37:22.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:22.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:22 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:24.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:24.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:24.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:24 smithi073 ceph-mon[66481]: pgmap v306: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-11T22:37:24.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:24.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:24.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:24 smithi099 ceph-mon[53203]: pgmap v306: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-11T22:37:25.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:25 smithi073 ceph-mon[66481]: pgmap v307: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-11T22:37:25.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:25 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:25 smithi099 ceph-mon[53203]: pgmap v307: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.0 KiB/s wr, 3 op/s 2024-03-11T22:37:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:25 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:26.940 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: Detected new or changed devices on smithi073 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi099.kagazk"]}]: dispatch 2024-03-11T22:37:26.941 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:26 smithi099 ceph-mon[53203]: Upgrade: It appears safe to stop mds.foofs.smithi099.kagazk 2024-03-11T22:37:27.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: Detected new or changed devices on smithi073 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi099.kagazk"]}]: dispatch 2024-03-11T22:37:27.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:26 smithi073 ceph-mon[66481]: Upgrade: It appears safe to stop mds.foofs.smithi099.kagazk 2024-03-11T22:37:27.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:27 smithi099 ceph-mon[53203]: pgmap v308: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-03-11T22:37:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:37:27.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:27 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:28.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:27 smithi073 ceph-mon[66481]: pgmap v308: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-03-11T22:37:28.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:28.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi099.kagazk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T22:37:28.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:27 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:28.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:28 smithi099 ceph-mon[53203]: Upgrade: Updating mds.foofs.smithi099.kagazk 2024-03-11T22:37:28.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:28 smithi099 ceph-mon[53203]: Deploying daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:37:29.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:28 smithi073 ceph-mon[66481]: Upgrade: Updating mds.foofs.smithi099.kagazk 2024-03-11T22:37:29.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:28 smithi073 ceph-mon[66481]: Deploying daemon mds.foofs.smithi099.kagazk on smithi099 2024-03-11T22:37:29.965 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:29 smithi099 ceph-mon[53203]: pgmap v309: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T22:37:29.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:29 smithi073 ceph-mon[66481]: pgmap v309: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T22:37:30.469 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:29 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[66477]: 2024-03-11T22:37:29.965+0000 7f6e2205e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:30.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:30.026+0000 7f962dd8c700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 8 2024-03-11T22:37:30.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:30.026+0000 7f9632d96700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 12 2024-03-11T22:37:30.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:30.027+0000 7f9630d92700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 15 2024-03-11T22:37:30.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:30.027+0000 7f962d58b700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 22 2024-03-11T22:37:30.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:37:30.027+0000 7f9631593700 -1 bdev(0x55ddd894c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 22 2024-03-11T22:37:31.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: Standby daemon mds.foofs.smithi073.lmopce assigned to filesystem foofs as rank 0 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-11T22:37:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:30 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:replay} 2024-03-11T22:37:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T22:37:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T22:37:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T22:37:31.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: Standby daemon mds.foofs.smithi073.lmopce assigned to filesystem foofs as rank 0 2024-03-11T22:37:31.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T22:37:31.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T22:37:31.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-11T22:37:31.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:30 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:replay} 2024-03-11T22:37:32.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:31 smithi073 ceph-mon[66481]: pgmap v311: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.0 MiB/s rd, 306 B/s wr, 3 op/s 2024-03-11T22:37:32.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:31 smithi099 ceph-mon[53203]: pgmap v311: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.0 MiB/s rd, 306 B/s wr, 3 op/s 2024-03-11T22:37:33.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:32 smithi073 ceph-mon[66481]: reconnect by client.14540 172.21.15.73:0/239032306 after 0 2024-03-11T22:37:33.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:32 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:reconnect 2024-03-11T22:37:33.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:32 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:reconnect} 2024-03-11T22:37:33.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:32 smithi073 ceph-mon[66481]: reconnect by client.24349 172.21.15.99:0/589926945 after 0.0309993 2024-03-11T22:37:33.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:32 smithi099 ceph-mon[53203]: reconnect by client.14540 172.21.15.73:0/239032306 after 0 2024-03-11T22:37:33.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:32 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:reconnect 2024-03-11T22:37:33.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:32 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:reconnect} 2024-03-11T22:37:33.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:32 smithi099 ceph-mon[53203]: reconnect by client.24349 172.21.15.99:0/589926945 after 0.0309993 2024-03-11T22:37:34.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:33 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:rejoin 2024-03-11T22:37:34.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:33 smithi099 ceph-mon[53203]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:rejoin} 2024-03-11T22:37:34.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:33 smithi099 ceph-mon[53203]: daemon mds.foofs.smithi073.lmopce is now active in filesystem foofs as rank 0 2024-03-11T22:37:34.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:33 smithi099 ceph-mon[53203]: pgmap v312: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 307 B/s wr, 4 op/s 2024-03-11T22:37:34.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:33 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:rejoin 2024-03-11T22:37:34.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:33 smithi073 ceph-mon[66481]: fsmap foofs:1/1 {0=foofs.smithi073.lmopce=up:rejoin} 2024-03-11T22:37:34.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:33 smithi073 ceph-mon[66481]: daemon mds.foofs.smithi073.lmopce is now active in filesystem foofs as rank 0 2024-03-11T22:37:34.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:33 smithi073 ceph-mon[66481]: pgmap v312: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 307 B/s wr, 4 op/s 2024-03-11T22:37:35.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:34 smithi073 ceph-mon[66481]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T22:37:35.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:34 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:active 2024-03-11T22:37:35.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:34 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 2024-03-11T22:37:35.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:34 smithi099 ceph-mon[53203]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T22:37:35.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:34 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635] up:active 2024-03-11T22:37:35.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:34 smithi099 ceph-mon[53203]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 2024-03-11T22:37:36.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:35 smithi073 ceph-mon[66481]: pgmap v313: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 307 B/s wr, 9 op/s 2024-03-11T22:37:36.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:35 smithi099 ceph-mon[53203]: pgmap v313: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 307 B/s wr, 9 op/s 2024-03-11T22:37:37.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:37 smithi099 ceph-mon[53203]: pgmap v314: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T22:37:38.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:37 smithi073 ceph-mon[66481]: pgmap v314: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T22:37:39.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:39.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:39.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:39.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:39.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:40.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T22:37:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: Cluster is now healthy 2024-03-11T22:37:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: mds.? [v2:172.21.15.99:6832/1412409555,v1:172.21.15.99:6833/1412409555] up:boot 2024-03-11T22:37:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:37:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:37:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:40 smithi099 ceph-mon[53203]: pgmap v315: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: Cluster is now healthy 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: mds.? [v2:172.21.15.99:6832/1412409555,v1:172.21.15.99:6833/1412409555] up:boot 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: fsmap foofs:1 {0=foofs.smithi073.lmopce=up:active} 1 up:standby 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:37:41.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:40 smithi073 ceph-mon[66481]: pgmap v315: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-03-11T22:37:42.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:41 smithi099 ceph-mon[53203]: pgmap v316: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.4 KiB/s wr, 12 op/s 2024-03-11T22:37:42.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:41 smithi073 ceph-mon[66481]: pgmap v316: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.4 KiB/s wr, 12 op/s 2024-03-11T22:37:43.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:43.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:43 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:43.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:43.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:43 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:44.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:44 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:44.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:44 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:44.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:44 smithi099 ceph-mon[53203]: pgmap v317: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-11T22:37:44.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:44 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:44.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:44 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:44.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:44 smithi073 ceph-mon[66481]: pgmap v317: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-03-11T22:37:46.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:45 smithi073 ceph-mon[66481]: pgmap v318: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-03-11T22:37:46.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:45 smithi099 ceph-mon[53203]: pgmap v318: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-03-11T22:37:48.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Detected new or changed devices on smithi099 2024-03-11T22:37:48.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all mds 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi073.lmopce"}]': finished 2024-03-11T22:37:48.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi099.kagazk"}]': finished 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all rgw 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all rbd-mirror 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all ceph-exporter 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all iscsi 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:48 smithi099 ceph-mon[53203]: pgmap v319: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Detected new or changed devices on smithi099 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all mds 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi073.lmopce"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi073.lmopce"}]': finished 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi099.kagazk"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi099.kagazk"}]': finished 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all rgw 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all rbd-mirror 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all ceph-exporter 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all iscsi 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:48.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:48 smithi073 ceph-mon[66481]: pgmap v319: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Upgrade: Updating nfs.foo.0.0.smithi073.gkqass (1/2) 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Creating key for client.nfs.foo.0.0.smithi073.gkqass 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:37:50.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Rados config object exists: conf-nfs.foo 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Creating key for client.nfs.foo.0.0.smithi073.gkqass-rgw 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Bind address in nfs.foo.0.0.smithi073.gkqass's ganesha conf is defaulting to empty 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: Deploying daemon nfs.foo.0.0.smithi073.gkqass on smithi073 2024-03-11T22:37:50.221 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:50 smithi073 ceph-mon[66481]: pgmap v320: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 7.1 KiB/s wr, 3 op/s 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Upgrade: Updating nfs.foo.0.0.smithi073.gkqass (1/2) 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Creating key for client.nfs.foo.0.0.smithi073.gkqass 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Rados config object exists: conf-nfs.foo 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Creating key for client.nfs.foo.0.0.smithi073.gkqass-rgw 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi073.gkqass-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Bind address in nfs.foo.0.0.smithi073.gkqass's ganesha conf is defaulting to empty 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: Deploying daemon nfs.foo.0.0.smithi073.gkqass on smithi073 2024-03-11T22:37:50.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:50 smithi099 ceph-mon[53203]: pgmap v320: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 7.1 KiB/s wr, 3 op/s 2024-03-11T22:37:52.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:51 smithi073 ceph-mon[66481]: pgmap v321: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.3 KiB/s wr, 3 op/s 2024-03-11T22:37:52.072 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:37:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:51 smithi099 ceph-mon[53203]: pgmap v321: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 7.3 KiB/s wr, 3 op/s 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (10m) 32s ago 19m 22.6M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (6m) 32s ago 19m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (6m) 10s ago 18m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (10m) 32s ago 19m 49.6M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (38s) 32s ago 13m 12.4M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:37:52.820 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (13s) 10s ago 13m 12.0M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (9m) 32s ago 20m 511M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (8m) 10s ago 18m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (8m) 32s ago 21m 59.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (7m) 10s ago 18m 46.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (13m) 32s ago 13m 269M - 4.0 e1d6a67b021e 2e7d67e7826a 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (13m) 10s ago 13m 86.4M - 4.0 e1d6a67b021e 3b96d29829f4 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (10m) 32s ago 19m 17.7M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (9m) 10s ago 18m 19.1M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (6m) 32s ago 18m 410M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (5m) 32s ago 17m 372M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (4m) 32s ago 17m 351M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (3m) 32s ago 16m 425M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (2m) 10s ago 16m 337M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:37:52.821 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (2m) 10s ago 15m 313M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:37:52.822 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (102s) 10s ago 15m 299M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:37:52.822 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (64s) 10s ago 15m 300M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:37:52.822 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (8m) 32s ago 18m 71.8M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:37:53.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:52 smithi073 ceph-mon[66481]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:53.115 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:52 smithi073 ceph-mon[66481]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:53.237 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:37:53.238 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:37:53.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:52 smithi099 ceph-mon[53203]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:53.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:52 smithi099 ceph-mon[53203]: from='client.34508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "mds", 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "osd", 2024-03-11T22:37:53.600 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "16/23 daemons upgraded", 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading nfs daemons", 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:37:53.601 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:37:54.013 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/4207022960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.73:0/239032306"}]: dispatch 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:54.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:53 smithi099 ceph-mon[53203]: pgmap v322: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-11T22:37:54.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/4207022960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.73:0/239032306"}]: dispatch 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:37:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:53 smithi073 ceph-mon[66481]: pgmap v322: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-11T22:37:55.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:54 smithi099 ceph-mon[53203]: Evicting (and blocklisting) client session 14540 (172.21.15.73:0/239032306) 2024-03-11T22:37:55.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:54 smithi099 ceph-mon[53203]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.73:0/239032306"}]': finished 2024-03-11T22:37:55.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:54 smithi099 ceph-mon[53203]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T22:37:55.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:54 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/984381160' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:37:55.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:54 smithi073 ceph-mon[66481]: Evicting (and blocklisting) client session 14540 (172.21.15.73:0/239032306) 2024-03-11T22:37:55.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:54 smithi073 ceph-mon[66481]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.73:0/239032306"}]': finished 2024-03-11T22:37:55.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:54 smithi073 ceph-mon[66481]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T22:37:55.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:54 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/984381160' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Upgrade: Updating nfs.foo.1.0.smithi099.ahqxhz (2/2) 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:37:56.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Rados config object exists: conf-nfs.foo 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz-rgw 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Bind address in nfs.foo.1.0.smithi099.ahqxhz's ganesha conf is defaulting to empty 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: Deploying daemon nfs.foo.1.0.smithi099.ahqxhz on smithi099 2024-03-11T22:37:56.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:56 smithi073 ceph-mon[66481]: pgmap v324: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 716 B/s wr, 1 op/s 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Upgrade: Updating nfs.foo.1.0.smithi099.ahqxhz (2/2) 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-11T22:37:56.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Rados config object exists: conf-nfs.foo 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Creating key for client.nfs.foo.1.0.smithi099.ahqxhz-rgw 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi099.ahqxhz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Bind address in nfs.foo.1.0.smithi099.ahqxhz's ganesha conf is defaulting to empty 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: Deploying daemon nfs.foo.1.0.smithi099.ahqxhz on smithi099 2024-03-11T22:37:56.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:56 smithi099 ceph-mon[53203]: pgmap v324: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 716 B/s wr, 1 op/s 2024-03-11T22:37:58.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:37:57 smithi099 ceph-mon[53203]: pgmap v325: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T22:37:58.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:37:57 smithi073 ceph-mon[66481]: pgmap v325: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T22:38:00.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:00.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:00.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:00 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:00.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:00 smithi073 ceph-mon[66481]: pgmap v326: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T22:38:00.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:00.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:00 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:00.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:00 smithi099 ceph-mon[53203]: pgmap v326: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T22:38:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:01 smithi073 ceph-mon[66481]: pgmap v327: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T22:38:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:01 smithi099 ceph-mon[53203]: pgmap v327: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T22:38:03.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:03 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:03.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:03 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:03.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:03 smithi073 ceph-mon[66481]: Evicting (and blocklisting) client session 24349 (172.21.15.99:0/589926945) 2024-03-11T22:38:03.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:03 smithi073 ceph-mon[66481]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.99:0/589926945"}]: dispatch 2024-03-11T22:38:03.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:03 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:03.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:03 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:03.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:03 smithi099 ceph-mon[53203]: Evicting (and blocklisting) client session 24349 (172.21.15.99:0/589926945) 2024-03-11T22:38:03.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:03 smithi099 ceph-mon[53203]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.99:0/589926945"}]: dispatch 2024-03-11T22:38:04.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:04 smithi073 ceph-mon[66481]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.99:0/589926945"}]': finished 2024-03-11T22:38:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:04 smithi073 ceph-mon[66481]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T22:38:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:04 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:04.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:04 smithi073 ceph-mon[66481]: pgmap v329: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-11T22:38:04.425 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:04 smithi099 ceph-mon[53203]: from='mds.? [v2:172.21.15.73:6834/3885494635,v1:172.21.15.73:6835/3885494635]' entity='mds.foofs.smithi073.lmopce' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.99:0/589926945"}]': finished 2024-03-11T22:38:04.425 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:04 smithi099 ceph-mon[53203]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T22:38:04.425 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:04.425 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:04 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:04.425 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:04 smithi099 ceph-mon[53203]: pgmap v329: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-11T22:38:05.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:05.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:05.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:05.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:06 smithi099 ceph-mon[53203]: pgmap v330: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-11T22:38:06.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:06 smithi073 ceph-mon[66481]: pgmap v330: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-11T22:38:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:08.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:07 smithi073 ceph-mon[66481]: pgmap v331: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-11T22:38:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:07 smithi099 ceph-mon[53203]: pgmap v331: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-11T22:38:09.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:09.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:10.848 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: pgmap v332: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.849 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi073.gkqass"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi073.gkqass"}]': finished 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi099.ahqxhz"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi099.ahqxhz"}]': finished 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:10.850 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:10 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: pgmap v332: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:38:11.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.244 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi073.gkqass"}]: dispatch 2024-03-11T22:38:11.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi073.gkqass"}]': finished 2024-03-11T22:38:11.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi099.ahqxhz"}]: dispatch 2024-03-11T22:38:11.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi099.ahqxhz"}]': finished 2024-03-11T22:38:11.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:11.245 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:10 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:11.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:11 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all nfs 2024-03-11T22:38:11.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:11 smithi073 ceph-mon[66481]: Upgrade: Setting container_image for all nvmeof 2024-03-11T22:38:11.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:11 smithi073 ceph-mon[66481]: Upgrade: Updating node-exporter.smithi073 (1/2) 2024-03-11T22:38:11.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:11 smithi073 ceph-mon[66481]: Deploying daemon node-exporter.smithi073 on smithi073 2024-03-11T22:38:11.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:11 smithi073 ceph-mon[66481]: pgmap v333: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:38:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:11 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all nfs 2024-03-11T22:38:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:11 smithi099 ceph-mon[53203]: Upgrade: Setting container_image for all nvmeof 2024-03-11T22:38:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:11 smithi099 ceph-mon[53203]: Upgrade: Updating node-exporter.smithi073 (1/2) 2024-03-11T22:38:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:11 smithi099 ceph-mon[53203]: Deploying daemon node-exporter.smithi073 on smithi073 2024-03-11T22:38:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:11 smithi099 ceph-mon[53203]: pgmap v333: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:38:14.089 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:13 smithi073 ceph-mon[66481]: pgmap v334: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 385 B/s wr, 0 op/s 2024-03-11T22:38:14.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:13 smithi099 ceph-mon[53203]: pgmap v334: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 385 B/s wr, 0 op/s 2024-03-11T22:38:16.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:15 smithi073 ceph-mon[66481]: pgmap v335: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:38:16.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:15 smithi099 ceph-mon[53203]: pgmap v335: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:38:18.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:17 smithi073 ceph-mon[66481]: pgmap v336: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:38:18.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:17 smithi099 ceph-mon[53203]: pgmap v336: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-11T22:38:19.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:19 smithi073 ceph-mon[66481]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:20.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:19 smithi099 ceph-mon[53203]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:22.219 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:21 smithi073 ceph-mon[66481]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:22.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:21 smithi099 ceph-mon[53203]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:24.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:24.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:24.365 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:38:25.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:24 smithi073 ceph-mon[66481]: pgmap v339: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:25.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:24 smithi073 ceph-mon[66481]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:25.030 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:24 smithi073 ceph-mon[66481]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:25.030 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:38:25.030 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (11m) 20s ago 20m 22.6M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:38:25.030 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (7m) 20s ago 20m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (7m) 22s ago 19m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (10m) 20s ago 19m 50.4M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (70s) 20s ago 14m 33.4M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (46s) 22s ago 14m 12.1M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (9m) 20s ago 21m 512M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (9m) 22s ago 19m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (8m) 20s ago 21m 60.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (8m) 22s ago 19m 47.5M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (32s) 20s ago 13m 61.3M - 5.7 a6477a474d71 71729d6d1bbe 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (26s) 22s ago 13m 17.0M - 5.7 a6477a474d71 0c5314ff8e49 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (10m) 20s ago 19m 17.4M - 1.3.1 1dbe0e931976 de84d98888a0 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (10m) 22s ago 19m 18.4M - 1.3.1 1dbe0e931976 7c016bb179fe 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (6m) 20s ago 18m 411M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (5m) 20s ago 18m 374M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (5m) 20s ago 17m 351M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (4m) 20s ago 17m 426M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:38:25.031 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (3m) 22s ago 16m 337M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:38:25.032 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (2m) 22s ago 16m 313M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:38:25.032 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (2m) 22s ago 16m 300M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:38:25.032 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (96s) 22s ago 15m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:38:25.032 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (9m) 20s ago 18m 71.8M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:38:25.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:24 smithi099 ceph-mon[53203]: pgmap v339: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:25.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:24 smithi099 ceph-mon[53203]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:25.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:24 smithi099 ceph-mon[53203]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:25.432 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:38:25.433 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:38:25.433 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:38:25.433 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:38:25.758 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "mds", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "osd", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "nfs", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "18/23 daemons upgraded", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-11T22:38:25.759 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:38:25.760 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:38:26.146 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:38:26.146 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:25 smithi073 ceph-mon[66481]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:26.146 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:25 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/113694615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:26.146 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:25 smithi073 ceph-mon[66481]: pgmap v340: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:26.146 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:25 smithi073 ceph-mon[66481]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:26.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:25 smithi099 ceph-mon[53203]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:25 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/113694615' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:25 smithi099 ceph-mon[53203]: pgmap v340: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:25 smithi099 ceph-mon[53203]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:27.109 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:26 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/197513884' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:38:27.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:26 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/197513884' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:38:28.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:27 smithi073 ceph-mon[66481]: pgmap v341: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:28.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:27 smithi099 ceph-mon[53203]: pgmap v341: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:29.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:29 smithi073 ceph-mon[66481]: pgmap v342: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:30.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:29 smithi099 ceph-mon[53203]: pgmap v342: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:32.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:32.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:32 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:32.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:32 smithi099 ceph-mon[53203]: pgmap v343: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:32.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:32.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:32 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:32.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:32 smithi073 ceph-mon[66481]: pgmap v343: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:34.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:33 smithi099 ceph-mon[53203]: Upgrade: Updating node-exporter.smithi099 (2/2) 2024-03-11T22:38:34.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:33 smithi099 ceph-mon[53203]: Deploying daemon node-exporter.smithi099 on smithi099 2024-03-11T22:38:34.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:33 smithi099 ceph-mon[53203]: pgmap v344: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:33 smithi073 ceph-mon[66481]: Upgrade: Updating node-exporter.smithi099 (2/2) 2024-03-11T22:38:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:33 smithi073 ceph-mon[66481]: Deploying daemon node-exporter.smithi099 on smithi099 2024-03-11T22:38:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:33 smithi073 ceph-mon[66481]: pgmap v344: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:36.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:35 smithi073 ceph-mon[66481]: pgmap v345: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:36.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:35 smithi099 ceph-mon[53203]: pgmap v345: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:38.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:37 smithi073 ceph-mon[66481]: pgmap v346: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:38.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:37 smithi099 ceph-mon[53203]: pgmap v346: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:39.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:39.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:39.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:39 smithi073 ceph-mon[66481]: pgmap v347: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:40.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:39 smithi099 ceph-mon[53203]: pgmap v347: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:42.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:41 smithi073 ceph-mon[66481]: pgmap v348: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:42.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:41 smithi099 ceph-mon[53203]: pgmap v348: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:44.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:43 smithi073 ceph-mon[66481]: pgmap v349: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:44.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:43 smithi099 ceph-mon[53203]: pgmap v349: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:46.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:45 smithi073 ceph-mon[66481]: pgmap v350: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:46.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:45 smithi099 ceph-mon[53203]: pgmap v350: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:47.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:47 smithi099 ceph-mon[53203]: pgmap v351: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:48.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:47 smithi073 ceph-mon[66481]: pgmap v351: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:48 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:49.195 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:49.195 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:49.195 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:49.196 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:49.196 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:48 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:50.197 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:49 smithi073 ceph-mon[66481]: pgmap v352: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:50.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:49 smithi099 ceph-mon[53203]: pgmap v352: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:52.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:52.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:52.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:51 smithi073 ceph-mon[66481]: pgmap v353: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:52.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:52.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:52.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:51 smithi099 ceph-mon[53203]: pgmap v353: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:53.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:53.187 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:52 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:53.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:53.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:52 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:54.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:54.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:54.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:54.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:54 smithi073 ceph-mon[66481]: pgmap v354: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:54.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:54.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:54.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:38:54.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:54 smithi099 ceph-mon[53203]: pgmap v354: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:38:55.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:55 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:55.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:55 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:56.502 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:38:56.692 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:56 smithi073 ceph-mon[66481]: pgmap v355: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:56.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:56 smithi099 ceph-mon[53203]: pgmap v355: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:57.170 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (11m) 3s ago 20m 23.1M - 0.23.0 ba2b418f427c bacfccf8cc59 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (7m) 3s ago 20m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (7m) 6s ago 20m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (11m) 3s ago 20m 51.0M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (102s) 3s ago 14m 33.8M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (78s) 6s ago 14m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (10m) 3s ago 21m 513M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (9m) 6s ago 19m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (9m) 3s ago 22m 61.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (8m) 6s ago 19m 47.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (64s) 3s ago 14m 62.1M - 5.7 a6477a474d71 71729d6d1bbe 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (58s) 6s ago 14m 61.2M - 5.7 a6477a474d71 0c5314ff8e49 2024-03-11T22:38:57.171 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (26s) 3s ago 20m 8497k - 1.5.0 0da6a335fe13 cc83e85a7c0c 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (9s) 6s ago 19m 3389k - 1.5.0 0da6a335fe13 6e5a58067d58 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (7m) 3s ago 19m 413M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (6m) 3s ago 18m 375M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (5m) 3s ago 18m 352M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (4m) 3s ago 17m 427M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (4m) 6s ago 17m 339M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (3m) 6s ago 16m 314M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (2m) 6s ago 16m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (2m) 6s ago 16m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:38:57.172 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (9m) 3s ago 19m 71.8M - 2.33.4 514e6a882f6e e593441e7107 2024-03-11T22:38:57.577 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:38:57.577 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:38:57.577 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:57.577 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:38:57.578 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:38:57.719 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:57 smithi073 ceph-mon[66481]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:57.916 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:38:57.916 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:38:57.916 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:38:57.916 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:38:57.916 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "mds", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "osd", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "nfs", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "18/23 daemons upgraded", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:38:57.917 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:38:57.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:57 smithi099 ceph-mon[53203]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:58.315 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:38:58.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:58.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:58.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/3799923136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: pgmap v356: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:58.882 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.883 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.884 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:58 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2426728091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/3799923136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: pgmap v356: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:38:58.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:58 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2426728091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:38:59.918 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:38:59 smithi073 ceph-mon[66481]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:38:59.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:38:59 smithi099 ceph-mon[53203]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:00.860 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:00 smithi073 ceph-mon[66481]: pgmap v357: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:00.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:00 smithi099 ceph-mon[53203]: pgmap v357: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:01 smithi073 ceph-mon[66481]: Upgrade: Updating prometheus.smithi073 2024-03-11T22:39:01.969 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:01 smithi073 ceph-mon[66481]: Deploying daemon prometheus.smithi073 on smithi073 2024-03-11T22:39:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:01 smithi099 ceph-mon[53203]: Upgrade: Updating prometheus.smithi073 2024-03-11T22:39:01.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:01 smithi099 ceph-mon[53203]: Deploying daemon prometheus.smithi073 on smithi073 2024-03-11T22:39:02.794 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:02 smithi073 ceph-mon[66481]: pgmap v358: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:02.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:02 smithi099 ceph-mon[53203]: pgmap v358: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:03.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:03 smithi073 ceph-mon[66481]: pgmap v359: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:03 smithi099 ceph-mon[53203]: pgmap v359: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:05.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:05 smithi073 ceph-mon[66481]: pgmap v360: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:05.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:05 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:05 smithi099 ceph-mon[53203]: pgmap v360: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:06.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:05 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:06.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:06 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:07.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:07.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:07.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:07.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:06 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:08.158 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:07 smithi073 ceph-mon[66481]: pgmap v361: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:08.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:07 smithi099 ceph-mon[53203]: pgmap v361: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:09.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:09.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:10.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:09 smithi073 ceph-mon[66481]: pgmap v362: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:10.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:09 smithi099 ceph-mon[53203]: pgmap v362: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:12.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:12.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:11 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:12.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:11 smithi073 ceph-mon[66481]: pgmap v363: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:11 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:11 smithi099 ceph-mon[53203]: pgmap v363: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:13.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:12 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:13.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:12 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:13.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:12 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:13.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:12 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:14.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:13 smithi073 ceph-mon[66481]: pgmap v364: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:14.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:13 smithi099 ceph-mon[53203]: pgmap v364: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.792 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.793 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:16 smithi073 ceph-mon[66481]: pgmap v365: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:16.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:16.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:16.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:16 smithi099 ceph-mon[53203]: pgmap v365: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:18.079 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:17 smithi073 ceph-mon[66481]: Upgrade: Updating alertmanager.smithi073 2024-03-11T22:39:18.080 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:17 smithi073 ceph-mon[66481]: Deploying daemon alertmanager.smithi073 on smithi073 2024-03-11T22:39:18.080 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:17 smithi073 ceph-mon[66481]: pgmap v366: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:18.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:17 smithi099 ceph-mon[53203]: Upgrade: Updating alertmanager.smithi073 2024-03-11T22:39:18.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:17 smithi099 ceph-mon[53203]: Deploying daemon alertmanager.smithi073 on smithi073 2024-03-11T22:39:18.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:17 smithi099 ceph-mon[53203]: pgmap v366: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:20.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:19 smithi073 ceph-mon[66481]: pgmap v367: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:20.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:19 smithi099 ceph-mon[53203]: pgmap v367: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:22.171 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:21 smithi073 ceph-mon[66481]: pgmap v368: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:21 smithi099 ceph-mon[53203]: pgmap v368: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:24.268 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:24.269 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:24.269 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:24.269 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:24.269 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:24 smithi073 ceph-mon[66481]: pgmap v369: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:24 smithi099 ceph-mon[53203]: pgmap v369: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:26.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:25 smithi073 ceph-mon[66481]: pgmap v370: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:25 smithi099 ceph-mon[53203]: pgmap v370: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:28.128 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:27 smithi073 ceph-mon[66481]: pgmap v371: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:28.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:27 smithi099 ceph-mon[53203]: pgmap v371: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:28.672 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (6s) 0s ago 21m 18.1M - 0.25.0 c8568f914cd2 c673167d93f4 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (8m) 0s ago 21m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (8m) 38s ago 20m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (11m) 0s ago 20m 51.1M - 8.3.5 dad864ee21e9 fdfa9b2e24cc 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (2m) 0s ago 15m 34.1M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (110s) 38s ago 15m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (10m) 0s ago 22m 515M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (10m) 38s ago 20m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (9m) 0s ago 22m 63.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:39:29.351 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (9m) 38s ago 20m 47.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (96s) 0s ago 14m 62.4M - 5.7 a6477a474d71 71729d6d1bbe 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (90s) 38s ago 14m 61.2M - 5.7 a6477a474d71 0c5314ff8e49 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (58s) 0s ago 20m 8292k - 1.5.0 0da6a335fe13 cc83e85a7c0c 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (41s) 38s ago 20m 3389k - 1.5.0 0da6a335fe13 6e5a58067d58 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (7m) 0s ago 19m 413M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (6m) 0s ago 19m 376M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (6m) 0s ago 18m 353M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (5m) 0s ago 18m 427M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (4m) 38s ago 17m 339M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (3m) 38s ago 17m 314M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (3m) 38s ago 17m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (2m) 38s ago 16m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:39:29.352 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (24s) 0s ago 19m 51.3M - 2.43.0 a07b618ecd1d 23a79623f55f 2024-03-11T22:39:29.630 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:29.630 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:29 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:29.630 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:29 smithi073 ceph-mon[66481]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:29.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:29.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:29 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:29.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:29 smithi099 ceph-mon[53203]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:39:29.758 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:39:29.759 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": true, 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [ 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "mds", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "osd", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "mon", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "nfs", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr", 2024-03-11T22:39:30.101 INFO:teuthology.orchestra.run.smithi073.stdout: "crash" 2024-03-11T22:39:30.102 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:39:30.102 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": "18/23 daemons upgraded", 2024-03-11T22:39:30.102 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "Currently upgrading alertmanager daemons", 2024-03-11T22:39:30.102 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:39:30.102 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:39:30.506 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/384795528' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: pgmap v372: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:30.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:30 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/3581691449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/384795528' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: pgmap v372: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:30 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/3581691449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:39:31.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:31 smithi073 ceph-mon[66481]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:31.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:31 smithi099 ceph-mon[53203]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:39:32.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:32 smithi073 ceph-mon[66481]: pgmap v373: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:32.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:32 smithi099 ceph-mon[53203]: pgmap v373: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:34.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.472 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:34 smithi073 ceph-mon[66481]: pgmap v374: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:34.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:34.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:34 smithi099 ceph-mon[53203]: pgmap v374: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:35.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:35 smithi073 ceph-mon[66481]: Upgrade: Updating grafana.smithi073 2024-03-11T22:39:35.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:35 smithi073 ceph-mon[66481]: Deploying daemon grafana.smithi073 on smithi073 2024-03-11T22:39:35.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:35 smithi073 ceph-mon[66481]: pgmap v375: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:36.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:35 smithi099 ceph-mon[53203]: Upgrade: Updating grafana.smithi073 2024-03-11T22:39:36.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:35 smithi099 ceph-mon[53203]: Deploying daemon grafana.smithi073 on smithi073 2024-03-11T22:39:36.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:35 smithi099 ceph-mon[53203]: pgmap v375: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:38.072 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:37 smithi073 ceph-mon[66481]: pgmap v376: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:38.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:37 smithi099 ceph-mon[53203]: pgmap v376: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:39.083 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:39.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:40.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:40.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:40.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:40 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:40.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:40 smithi073 ceph-mon[66481]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:40 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:40.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:40 smithi099 ceph-mon[53203]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:42.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:41 smithi073 ceph-mon[66481]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:42.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:41 smithi099 ceph-mon[53203]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:44.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:43 smithi073 ceph-mon[66481]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:44.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:43 smithi099 ceph-mon[53203]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:46.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:45 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:46.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:45 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:46.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:45 smithi099 ceph-mon[53203]: pgmap v380: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:46.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:45 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:46.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:45 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:46.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:45 smithi073 ceph-mon[66481]: pgmap v380: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:47.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:47 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:47.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:47 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:47 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:47 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:48.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:48 smithi073 ceph-mon[66481]: pgmap v381: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:48.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:48 smithi099 ceph-mon[53203]: pgmap v381: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:50.720 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.721 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.722 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: Upgrade: Finalizing container_image settings 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-11T22:39:50.723 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.724 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: Upgrade: Complete! 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.725 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.726 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:50 smithi073 ceph-mon[66481]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:50.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: Upgrade: Finalizing container_image settings 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-11T22:39:50.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: Upgrade: Complete! 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:50.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:50 smithi099 ceph-mon[53203]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:52.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:51 smithi073 ceph-mon[66481]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:52.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:51 smithi099 ceph-mon[53203]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:54.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:53 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:54.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:53 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:39:55.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:54 smithi073 ceph-mon[66481]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:55.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:55.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:54 smithi099 ceph-mon[53203]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:39:55.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:39:57.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:56 smithi073 ceph-mon[66481]: pgmap v385: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:57.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:56 smithi099 ceph-mon[53203]: pgmap v385: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:58.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:57 smithi073 ceph-mon[66481]: pgmap v386: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:58.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:57 smithi099 ceph-mon[53203]: pgmap v386: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:39:59.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:39:59 smithi073 ceph-mon[66481]: pgmap v387: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:00.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:39:59 smithi099 ceph-mon[53203]: pgmap v387: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:01.145 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:00 smithi073 ceph-mon[66481]: overall HEALTH_OK 2024-03-11T22:40:01.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:00 smithi099 ceph-mon[53203]: overall HEALTH_OK 2024-03-11T22:40:01.435 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch ps' 2024-03-11T22:40:02.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:01 smithi073 ceph-mon[66481]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:40:02.062 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:01 smithi073 ceph-mon[66481]: pgmap v388: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:02.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:01 smithi099 ceph-mon[53203]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:40:02.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:01 smithi099 ceph-mon[53203]: pgmap v388: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (40s) 18s ago 21m 18.1M - 0.25.0 c8568f914cd2 c673167d93f4 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (8m) 18s ago 21m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (8m) 72s ago 21m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (24s) 18s ago 21m 51.5M - 9.4.12 a0ea5e3c46d9 92bea2040024 2024-03-11T22:40:03.347 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (2m) 18s ago 15m 34.2M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (2m) 72s ago 15m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (11m) 18s ago 23m 515M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (10m) 72s ago 21m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (10m) 18s ago 23m 63.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (9m) 72s ago 21m 47.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (2m) 18s ago 15m 62.5M - 5.7 a6477a474d71 71729d6d1bbe 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (2m) 72s ago 15m 61.2M - 5.7 a6477a474d71 0c5314ff8e49 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (92s) 18s ago 21m 7914k - 1.5.0 0da6a335fe13 cc83e85a7c0c 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (75s) 72s ago 20m 3389k - 1.5.0 0da6a335fe13 6e5a58067d58 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (8m) 18s ago 20m 414M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (7m) 18s ago 19m 376M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (6m) 18s ago 19m 353M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (5m) 18s ago 18m 427M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:40:03.348 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (5m) 72s ago 18m 339M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:40:03.349 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (4m) 72s ago 18m 314M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:40:03.349 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (3m) 72s ago 17m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:40:03.349 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (3m) 72s ago 17m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:40:03.349 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (58s) 18s ago 20m 51.4M - 2.43.0 a07b618ecd1d 23a79623f55f 2024-03-11T22:40:03.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:03 smithi073 ceph-mon[66481]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:40:03.961 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:03 smithi073 ceph-mon[66481]: pgmap v389: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:03.985 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions' 2024-03-11T22:40:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:03 smithi099 ceph-mon[53203]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:40:04.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:03 smithi099 ceph-mon[53203]: pgmap v389: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:06.011 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:40:06.012 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:40:06.013 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:40:06.013 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:40:06.013 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:40:06.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:05 smithi073 ceph-mon[66481]: pgmap v390: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:06.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:05 smithi099 ceph-mon[53203]: pgmap v390: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:06.836 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-11T22:40:07.147 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:06 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/4264032150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:40:07.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:06 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/4264032150' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:40:08.131 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:07 smithi073 ceph-mon[66481]: pgmap v391: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:08.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:07 smithi099 ceph-mon[53203]: pgmap v391: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:08.436 INFO:teuthology.orchestra.run.smithi073.stdout:wait for servicemap items w/ changing names to refresh 2024-03-11T22:40:09.030 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'sleep 60' 2024-03-11T22:40:09.054 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:08 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:09.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:08 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:09.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:09 smithi073 ceph-mon[66481]: pgmap v392: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:10.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:09 smithi099 ceph-mon[53203]: pgmap v392: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:12.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:11 smithi073 ceph-mon[66481]: pgmap v393: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:12.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:11 smithi099 ceph-mon[53203]: pgmap v393: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:14.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:13 smithi073 ceph-mon[66481]: pgmap v394: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:14.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:13 smithi099 ceph-mon[53203]: pgmap v394: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:16.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:15 smithi073 ceph-mon[66481]: pgmap v395: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:16.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:15 smithi099 ceph-mon[53203]: pgmap v395: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:18.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:17 smithi073 ceph-mon[66481]: pgmap v396: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:18.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:17 smithi099 ceph-mon[53203]: pgmap v396: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:19.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:19 smithi073 ceph-mon[66481]: pgmap v397: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:20.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:19 smithi099 ceph-mon[53203]: pgmap v397: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:22.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:21 smithi073 ceph-mon[66481]: pgmap v398: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:22.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:21 smithi099 ceph-mon[53203]: pgmap v398: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:24.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:23 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:24.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:23 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:25.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:24 smithi073 ceph-mon[66481]: pgmap v399: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:25.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:24 smithi099 ceph-mon[53203]: pgmap v399: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:26.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:25 smithi073 ceph-mon[66481]: pgmap v400: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:26.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:25 smithi099 ceph-mon[53203]: pgmap v400: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:28.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:27 smithi073 ceph-mon[66481]: pgmap v401: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:28.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:27 smithi099 ceph-mon[53203]: pgmap v401: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:30 smithi073 ceph-mon[66481]: pgmap v402: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:31.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:30 smithi099 ceph-mon[53203]: pgmap v402: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:33.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:32 smithi073 ceph-mon[66481]: pgmap v403: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:33.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:32 smithi099 ceph-mon[53203]: pgmap v403: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:35.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:34 smithi073 ceph-mon[66481]: pgmap v404: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:35.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:34 smithi099 ceph-mon[53203]: pgmap v404: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:37.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:36 smithi073 ceph-mon[66481]: pgmap v405: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:37.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:36 smithi099 ceph-mon[53203]: pgmap v405: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:39.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:38 smithi073 ceph-mon[66481]: pgmap v406: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:39.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:38 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:38 smithi099 ceph-mon[53203]: pgmap v406: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:38 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:41.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:40 smithi073 ceph-mon[66481]: pgmap v407: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:41.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:40 smithi099 ceph-mon[53203]: pgmap v407: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:43.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:43 smithi073 ceph-mon[66481]: pgmap v408: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:43.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:43 smithi099 ceph-mon[53203]: pgmap v408: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:45.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:45 smithi073 ceph-mon[66481]: pgmap v409: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:45.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:45 smithi099 ceph-mon[53203]: pgmap v409: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:47.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:47 smithi073 ceph-mon[66481]: pgmap v410: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:47 smithi099 ceph-mon[53203]: pgmap v410: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:49.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:49 smithi073 ceph-mon[66481]: pgmap v411: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:49.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:49 smithi099 ceph-mon[53203]: pgmap v411: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:50.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:50 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:40:50.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:50 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:40:51.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:51 smithi073 ceph-mon[66481]: pgmap v412: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:51.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:40:51.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:40:51.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:40:51.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:51 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:40:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:51 smithi099 ceph-mon[53203]: pgmap v412: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:40:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:40:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:40:51.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:51 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' 2024-03-11T22:40:53.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:53 smithi073 ceph-mon[66481]: pgmap v413: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:53.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:53 smithi099 ceph-mon[53203]: pgmap v413: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:54.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:54 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:54.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:54 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:40:55.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:55 smithi073 ceph-mon[66481]: pgmap v414: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:55.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:55 smithi099 ceph-mon[53203]: pgmap v414: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:40:57.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:57 smithi073 ceph-mon[66481]: pgmap v415: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:57.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:57 smithi099 ceph-mon[53203]: pgmap v415: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:59.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:40:59 smithi073 ceph-mon[66481]: pgmap v416: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:40:59.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:40:59 smithi099 ceph-mon[53203]: pgmap v416: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:01.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:01 smithi073 ceph-mon[66481]: pgmap v417: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:01 smithi099 ceph-mon[53203]: pgmap v417: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:03.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:03 smithi073 ceph-mon[66481]: pgmap v418: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:03.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:03 smithi099 ceph-mon[53203]: pgmap v418: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:05.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:05 smithi073 ceph-mon[66481]: pgmap v419: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:05.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:05 smithi099 ceph-mon[53203]: pgmap v419: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:07.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:07 smithi073 ceph-mon[66481]: pgmap v420: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:07.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:07 smithi099 ceph-mon[53203]: pgmap v420: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:09.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:09 smithi073 ceph-mon[66481]: pgmap v421: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:09.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:09 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:09.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:09 smithi099 ceph-mon[53203]: pgmap v421: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:09.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:09 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:11.162 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:11 smithi073 ceph-mon[66481]: pgmap v422: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:11.192 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch ps' 2024-03-11T22:41:11.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:11 smithi099 ceph-mon[53203]: pgmap v422: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:13.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:13 smithi073 ceph-mon[66481]: pgmap v423: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:13.286 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:alertmanager.smithi073 smithi073 *:9093,9094 running (110s) 88s ago 23m 18.1M - 0.25.0 c8568f914cd2 c673167d93f4 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi073 smithi073 running (10m) 88s ago 22m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 24907cde049a 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:crash.smithi099 smithi099 running (9m) 2m ago 22m 6660k - 19.0.0-1607-g94a53970 a6477a474d71 446dda51037b 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:grafana.smithi073 smithi073 *:3000 running (94s) 88s ago 22m 51.5M - 9.4.12 a0ea5e3c46d9 92bea2040024 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi073.lmopce smithi073 running (3m) 88s ago 17m 34.2M - 19.0.0-1607-g94a53970 a6477a474d71 40ebb44dc3d1 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mds.foofs.smithi099.kagazk smithi099 running (3m) 2m ago 16m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 80ff0539d723 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi073.pglkvm smithi073 *:8443,9283,8765 running (12m) 88s ago 24m 515M - 19.0.0-1607-g94a53970 a6477a474d71 2056d49e3b8f 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.smithi099.rcoixq smithi099 *:8443,9283,8765 running (11m) 2m ago 22m 452M - 19.0.0-1607-g94a53970 a6477a474d71 67189ca588d6 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi073 smithi073 running (11m) 88s ago 24m 63.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d9d041e225ca 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:mon.smithi099 smithi099 running (11m) 2m ago 22m 47.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 bd2a492ac8bb 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.0.0.smithi073.gkqass smithi073 *:2049 running (3m) 88s ago 16m 62.5M - 5.7 a6477a474d71 71729d6d1bbe 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:nfs.foo.1.0.smithi099.ahqxhz smithi099 *:2049 running (3m) 2m ago 16m 61.2M - 5.7 a6477a474d71 0c5314ff8e49 2024-03-11T22:41:13.287 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi073 smithi073 *:9100 running (2m) 88s ago 22m 7914k - 1.5.0 0da6a335fe13 cc83e85a7c0c 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:node-exporter.smithi099 smithi099 *:9100 running (2m) 2m ago 22m 3389k - 1.5.0 0da6a335fe13 6e5a58067d58 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (9m) 88s ago 21m 414M 1936M 19.0.0-1607-g94a53970 a6477a474d71 c9b79857231e 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi073 running (8m) 88s ago 20m 376M 1936M 19.0.0-1607-g94a53970 a6477a474d71 955cf0b05451 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi073 running (7m) 88s ago 20m 353M 1936M 19.0.0-1607-g94a53970 a6477a474d71 4500cd7061f9 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (7m) 88s ago 20m 427M 1936M 19.0.0-1607-g94a53970 a6477a474d71 810f2752eb26 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi099 running (6m) 2m ago 19m 339M 2704M 19.0.0-1607-g94a53970 a6477a474d71 97e1ed6508c3 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi099 running (5m) 2m ago 19m 314M 2704M 19.0.0-1607-g94a53970 a6477a474d71 33ae24167520 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi099 running (5m) 2m ago 18m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 e7b52a9fb498 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi099 running (4m) 2m ago 18m 301M 2704M 19.0.0-1607-g94a53970 a6477a474d71 f3cd6f3bf5ea 2024-03-11T22:41:13.288 INFO:teuthology.orchestra.run.smithi073.stdout:prometheus.smithi073 smithi073 *:9095 running (2m) 88s ago 21m 51.4M - 2.43.0 a07b618ecd1d 23a79623f55f 2024-03-11T22:41:13.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:13 smithi099 ceph-mon[53203]: pgmap v423: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:13.968 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch upgrade status' 2024-03-11T22:41:14.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:14 smithi073 ceph-mon[66481]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:14.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:14 smithi099 ceph-mon[53203]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:15.217 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:15 smithi073 ceph-mon[66481]: pgmap v424: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:15.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:15 smithi099 ceph-mon[53203]: pgmap v424: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "target_image": null, 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "in_progress": false, 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "which": "", 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "services_complete": [], 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "progress": null, 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "message": "", 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout: "is_paused": false 2024-03-11T22:41:15.918 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:41:16.726 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph health detail' 2024-03-11T22:41:17.310 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:17 smithi073 ceph-mon[66481]: pgmap v425: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:17.311 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:17 smithi073 ceph-mon[66481]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:17.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:17 smithi099 ceph-mon[53203]: pgmap v425: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:17.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:17 smithi099 ceph-mon[53203]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:18.690 INFO:teuthology.orchestra.run.smithi073.stdout:HEALTH_OK 2024-03-11T22:41:19.263 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:19 smithi073 ceph-mon[66481]: pgmap v426: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:19.263 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:19 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/1562628870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:41:19.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions' 2024-03-11T22:41:19.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:19 smithi099 ceph-mon[53203]: pgmap v426: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:19.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:19 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/1562628870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-11T22:41:21.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:21 smithi073 ceph-mon[66481]: pgmap v427: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:21.347 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "mon": { 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr": { 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "osd": { 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 8 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "mds": { 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 2 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:41:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: "overall": { 2024-03-11T22:41:21.349 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)": 14 2024-03-11T22:41:21.349 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:41:21.349 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-11T22:41:21.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:21 smithi099 ceph-mon[53203]: pgmap v427: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:22.113 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-11T22:41:22.423 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:22 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2883172758' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:22.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:22 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2883172758' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:23.427 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:23 smithi073 ceph-mon[66481]: pgmap v428: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:23.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:23 smithi099 ceph-mon[53203]: pgmap v428: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:true 2024-03-11T22:41:24.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:24 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:24.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:24 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2864429242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:24 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:24.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:24 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2864429242' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:24.657 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-11T22:41:25.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:25 smithi073 ceph-mon[66481]: pgmap v429: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:25.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:25 smithi099 ceph-mon[53203]: pgmap v429: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)" 2024-03-11T22:41:27.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:27 smithi073 ceph-mon[66481]: pgmap v430: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:27.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:27 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/2953114919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:27.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:27 smithi099 ceph-mon[53203]: pgmap v430: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:27.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:27 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/2953114919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T22:41:27.539 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-03-11T22:41:27.540 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-03-11T22:41:27.540 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- ceph orch ls -f json 2024-03-11T22:41:29.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:29 smithi073 ceph-mon[66481]: pgmap v431: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:29.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:29 smithi099 ceph-mon[53203]: pgmap v431: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:29.540 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-11T22:41:29.540 INFO:teuthology.orchestra.run.smithi073.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-11T22:17:33.673110Z", "last_refresh": "2024-03-11T22:39:44.967665Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-11T22:17:28.345495Z", "last_refresh": "2024-03-11T22:38:50.852437Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-03-11T22:17:31.125131Z", "last_refresh": "2024-03-11T22:39:44.967837Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-11T22:24:06.925899Z", "last_refresh": "2024-03-11T22:38:50.853095Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-11T22:17:26.767881Z", "last_refresh": "2024-03-11T22:38:50.852566Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi073:172.21.15.73=smithi073", "smithi099:172.21.15.99=smithi099"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-11T22:18:16.830889Z", "last_refresh": "2024-03-11T22:38:50.852644Z", "running": 2, "size": 2}}, {"events": ["2024-03-11T22:40:50.404983Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-03-11T22:24:28.047774Z", "last_refresh": "2024-03-11T22:38:50.853182Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-11T22:17:32.323562Z", "last_refresh": "2024-03-11T22:38:50.852718Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", "last_refresh": "2024-03-11T22:38:50.852801Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-11T22:17:29.791459Z", "last_refresh": "2024-03-11T22:39:44.967988Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-11T22:41:30.212 INFO:tasks.cephadm:nfs.foo has 2/2 2024-03-11T22:41:30.213 DEBUG:teuthology.parallel:result is None 2024-03-11T22:41:30.213 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-11T22:41:30.224 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:41:30.224 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-03-11T22:41:30.250 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:30 smithi073 ceph-mon[66481]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:41:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:+ umount /mnt/foo 2024-03-11T22:41:30.265 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:41:30.273 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:41:30.274 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-03-11T22:41:30.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:30 smithi099 ceph-mon[53203]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:41:31.220 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:31 smithi073 ceph-mon[66481]: pgmap v432: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:31.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:31 smithi099 ceph-mon[53203]: pgmap v432: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:32.235 INFO:teuthology.orchestra.run.smithi073.stdout: "foo" 2024-03-11T22:41:32.836 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-03-11T22:41:33.389 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:33 smithi073 ceph-mon[66481]: pgmap v433: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:33.390 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:33 smithi073 ceph-mon[66481]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:33.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:33 smithi099 ceph-mon[53203]: pgmap v433: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:33.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:33 smithi099 ceph-mon[53203]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:34.846 INFO:teuthology.orchestra.run.smithi073.stdout:[ 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "access_type": "RW", 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "clients": [], 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "cluster_id": "foo", 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "export_id": 1, 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "fsal": { 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "fs_name": "foofs", 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "CEPH", 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: "user_id": "nfs.foo.1" 2024-03-11T22:41:34.847 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "protocols": [ 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: 4 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "pseudo": "/fake", 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "security_label": true, 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "squash": "none", 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "transports": [ 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: "TCP" 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-11T22:41:34.848 INFO:teuthology.orchestra.run.smithi073.stdout:] 2024-03-11T22:41:35.408 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:35 smithi073 ceph-mon[66481]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:35.435 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-03-11T22:41:35.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:35 smithi099 ceph-mon[53203]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:36.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:36 smithi073 ceph-mon[66481]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:36.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:36 smithi099 ceph-mon[53203]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:41:37.269 INFO:teuthology.orchestra.run.smithi073.stdout:foo rec-0000000000000004:nfs.foo.1 2024-03-11T22:41:37.269 INFO:teuthology.orchestra.run.smithi073.stdout:foo grace 2024-03-11T22:41:37.269 INFO:teuthology.orchestra.run.smithi073.stdout:foo rec-0000000000000004:nfs.foo.0 2024-03-11T22:41:37.269 INFO:teuthology.orchestra.run.smithi073.stdout:foo conf-nfs.foo 2024-03-11T22:41:37.269 INFO:teuthology.orchestra.run.smithi073.stdout:foo export-1 2024-03-11T22:41:37.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:37 smithi099 ceph-mon[53203]: pgmap v435: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:37.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:37 smithi073 ceph-mon[66481]: pgmap v435: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T22:41:38.128 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:41:38.138 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:41:38.138 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 -- bash -c 'set -ex 2024-03-11T22:41:38.138 DEBUG:teuthology.orchestra.run.smithi073:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-03-11T22:41:38.138 DEBUG:teuthology.orchestra.run.smithi073:> ' 2024-03-11T22:41:38.452 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:38 smithi073 ceph-mon[66481]: mgrmap e42: smithi073.pglkvm(active, since 11m), standbys: smithi099.rcoixq 2024-03-11T22:41:38.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:38 smithi099 ceph-mon[53203]: mgrmap e42: smithi073.pglkvm(active, since 11m), standbys: smithi099.rcoixq 2024-03-11T22:41:39.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:39 smithi073 ceph-mon[66481]: pgmap v436: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:39.446 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:39 smithi073 ceph-mon[66481]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:39 smithi099 ceph-mon[53203]: pgmap v436: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T22:41:39.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:39 smithi099 ceph-mon[53203]: from='mgr.34104 172.21.15.73:0/1080205588' entity='mgr.smithi073.pglkvm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:41:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-03-11T22:41:40.146 INFO:teuthology.orchestra.run.smithi073.stderr:+ [[ 6 -gt 2 ]] 2024-03-11T22:41:40.778 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:41:40.789 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:41:40.790 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:41:40.820 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:41:40.856 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:41:40.856 DEBUG:teuthology.orchestra.run.smithi073:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:41:40.875 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:41:40.914 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:41:40.915 INFO:tasks.cephadm.mon.smithi073:Stopping mon.smithi073... 2024-03-11T22:41:40.915 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073 2024-03-11T22:41:41.128 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:40 smithi073 systemd[1]: Stopping Ceph mon.smithi073 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:41:41.470 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 ceph-mon[66481]: pgmap v437: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-11T22:41:41.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 ceph-mon[66481]: from='client.? 172.21.15.73:0/959824172' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-11T22:41:41.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[66477]: 2024-03-11T22:41:41.233+0000 7f6e27869700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi073 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:41:41.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073[66477]: 2024-03-11T22:41:41.233+0000 7f6e27869700 -1 mon.smithi073@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:41:41.471 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 podman[112395]: 2024-03-11 22:41:41.315515409 +0000 UTC m=+0.219651128 container died d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:41:41.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:41 smithi099 ceph-mon[53203]: pgmap v437: 97 pgs: 97 active+clean; 222 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-11T22:41:41.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:41 smithi099 ceph-mon[53203]: from='client.? 172.21.15.73:0/959824172' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-11T22:41:41.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 podman[112395]: 2024-03-11 22:41:41.666947571 +0000 UTC m=+0.571083290 container cleanup d9d041e225caa35693ef9c2fae8b0ebb1363c054840f9204c75f3267c68000a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T22:41:41.970 INFO:journalctl@ceph.mon.smithi073.smithi073.stdout:Mar 11 22:41:41 smithi073 bash[112395]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi073 2024-03-11T22:41:42.147 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi073.service' 2024-03-11T22:41:42.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:41:42.207 INFO:tasks.cephadm.mon.smithi073:Stopped mon.smithi073 2024-03-11T22:41:42.207 INFO:tasks.cephadm.mon.smithi099:Stopping mon.smithi099... 2024-03-11T22:41:42.207 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi099 2024-03-11T22:41:42.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:42 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:41:42.974 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099[53198]: 2024-03-11T22:41:42.524+0000 7f926419b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:41:42.974 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099[53198]: 2024-03-11T22:41:42.524+0000 7f926419b700 -1 mon.smithi099@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:41:42.974 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:42 smithi099 podman[78745]: 2024-03-11 22:41:42.678365289 +0000 UTC m=+0.285379285 container died bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T22:41:43.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:43 smithi099 podman[78745]: 2024-03-11 22:41:43.158734571 +0000 UTC m=+0.765748609 container cleanup bd2a492ac8bbf993e6a77859bddc5f64934bfeea5ab8e20b836d01a4a55b94bb (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T22:41:43.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Mar 11 22:41:43 smithi099 bash[78745]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-mon-smithi099 2024-03-11T22:41:43.580 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@mon.smithi099.service' 2024-03-11T22:41:43.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:41:43.630 INFO:tasks.cephadm.mon.smithi099:Stopped mon.smithi099 2024-03-11T22:41:43.631 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:41:43.631 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:41:43.937 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:43 smithi073 systemd[1]: Stopping Ceph osd.0 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:41:44.220 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:43 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:41:43.929+0000 7f36bd219700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:41:44.220 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:43 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:41:43.929+0000 7f36bd219700 -1 osd.0 155 *** Got signal Terminated *** 2024-03-11T22:41:44.220 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:43 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0[72543]: 2024-03-11T22:41:43.929+0000 7f36bd219700 -1 osd.0 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:41:49.839 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:49 smithi073 podman[112531]: 2024-03-11 22:41:49.540053731 +0000 UTC m=+5.717267489 container died c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:41:50.180 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:49 smithi073 podman[112531]: 2024-03-11 22:41:49.922190888 +0000 UTC m=+6.099404653 container cleanup c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:41:50.180 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:49 smithi073 bash[112531]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:41:50.470 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112582]: 2024-03-11 22:41:50.173535543 +0000 UTC m=+0.630701255 container remove c9b79857231e04ac31b4c36d63eaee7ed8fa3f2879cb51016041d87dee952ba7 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T22:41:50.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112677]: 2024-03-11 22:41:50.376973876 +0000 UTC m=+0.018140152 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:41:50.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112677]: 2024-03-11 22:41:50.502374412 +0000 UTC m=+0.143540717 container create 8750e5709561f4ecd2558e86f4d3e6459b0b6c10bd5b811176a12cd5d66cb92a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T22:41:50.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112677]: 2024-03-11 22:41:50.702514694 +0000 UTC m=+0.343680965 container init 8750e5709561f4ecd2558e86f4d3e6459b0b6c10bd5b811176a12cd5d66cb92a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:41:50.809 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112677]: 2024-03-11 22:41:50.705830297 +0000 UTC m=+0.346996571 container start 8750e5709561f4ecd2558e86f4d3e6459b0b6c10bd5b811176a12cd5d66cb92a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:41:50.810 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:50 smithi073 podman[112677]: 2024-03-11 22:41:50.802252355 +0000 UTC m=+0.443418624 container attach 8750e5709561f4ecd2558e86f4d3e6459b0b6c10bd5b811176a12cd5d66cb92a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:41:51.220 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 11 22:41:51 smithi073 podman[112677]: 2024-03-11 22:41:51.0020853 +0000 UTC m=+0.643251567 container died 8750e5709561f4ecd2558e86f4d3e6459b0b6c10bd5b811176a12cd5d66cb92a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:41:51.614 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:41:51.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:41:51.657 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:41:51.657 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:41:51.657 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:41:51.970 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:51 smithi073 systemd[1]: Stopping Ceph osd.1 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:41:52.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:52 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:41:52.017+0000 7f964a7ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:41:52.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:52 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:41:52.017+0000 7f964a7ca700 -1 osd.1 155 *** Got signal Terminated *** 2024-03-11T22:41:52.470 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:52 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1[75222]: 2024-03-11T22:41:52.017+0000 7f964a7ca700 -1 osd.1 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:41:57.935 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:57 smithi073 podman[112794]: 2024-03-11 22:41:57.615219786 +0000 UTC m=+5.713153394 container died 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:41:58.220 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112794]: 2024-03-11 22:41:58.009358449 +0000 UTC m=+6.107292052 container cleanup 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:41:58.220 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 bash[112794]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:41:58.551 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112813]: 2024-03-11 22:41:58.244040297 +0000 UTC m=+0.628179330 container remove 955cf0b05451c71fa04c48d0e122389fce299b146ac664194111afac5ef44a90 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T22:41:58.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112906]: 2024-03-11 22:41:58.544901531 +0000 UTC m=+0.114163209 container create 3afe69cb7e08526c382bff384c424471f14712d1b2a1840cc0e9e3d81f19cd40 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T22:41:58.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112906]: 2024-03-11 22:41:58.44876963 +0000 UTC m=+0.018031313 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:41:58.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112906]: 2024-03-11 22:41:58.73914755 +0000 UTC m=+0.308409234 container init 3afe69cb7e08526c382bff384c424471f14712d1b2a1840cc0e9e3d81f19cd40 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:41:58.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112906]: 2024-03-11 22:41:58.742706721 +0000 UTC m=+0.311968409 container start 3afe69cb7e08526c382bff384c424471f14712d1b2a1840cc0e9e3d81f19cd40 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:41:58.821 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:58 smithi073 podman[112906]: 2024-03-11 22:41:58.81379001 +0000 UTC m=+0.383051699 container attach 3afe69cb7e08526c382bff384c424471f14712d1b2a1840cc0e9e3d81f19cd40 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:41:59.220 INFO:journalctl@ceph.osd.1.smithi073.stdout:Mar 11 22:41:59 smithi073 podman[112906]: 2024-03-11 22:41:59.038699402 +0000 UTC m=+0.607961081 container died 3afe69cb7e08526c382bff384c424471f14712d1b2a1840cc0e9e3d81f19cd40 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-11T22:41:59.614 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:41:59.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:41:59.657 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:41:59.657 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:41:59.657 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:41:59.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:41:59 smithi073 systemd[1]: Stopping Ceph osd.2 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:00.470 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:41:59 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:41:59.995+0000 7fb5023fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:00.470 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:41:59 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:41:59.995+0000 7fb5023fb700 -1 osd.2 155 *** Got signal Terminated *** 2024-03-11T22:42:00.470 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:41:59 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2[77776]: 2024-03-11T22:41:59.995+0000 7fb5023fb700 -1 osd.2 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:05.654 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:05 smithi073 podman[113010]: 2024-03-11 22:42:05.371012638 +0000 UTC m=+5.490501362 container died 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T22:42:05.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:05 smithi073 podman[113010]: 2024-03-11 22:42:05.795341147 +0000 UTC m=+5.914829871 container cleanup 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:42:05.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:05 smithi073 bash[113010]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:42:06.400 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:05 smithi073 podman[113029]: 2024-03-11 22:42:05.988362069 +0000 UTC m=+0.616986266 container remove 4500cd7061f9fac42a7eaeb71abce63393367fc3664063225fe96e2e6559893f (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-03-11T22:42:06.708 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.296841545 +0000 UTC m=+0.018774556 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:06.708 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.397519228 +0000 UTC m=+0.119452271 container create 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T22:42:06.708 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.625195125 +0000 UTC m=+0.347128116 container init 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:06.708 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.628211148 +0000 UTC m=+0.350144143 container start 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:42:06.708 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.700121319 +0000 UTC m=+0.422054322 container attach 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T22:42:06.970 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:06 smithi073 podman[113123]: 2024-03-11 22:42:06.91757858 +0000 UTC m=+0.639511581 container died 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T22:42:07.516 INFO:journalctl@ceph.osd.2.smithi073.stdout:Mar 11 22:42:07 smithi073 podman[113150]: 2024-03-11 22:42:07.495432772 +0000 UTC m=+0.572914054 container remove 1885f7f087e9ecb4522814bed629c596ba8dfd023738e619951715df86b3b317 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T22:42:07.522 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:42:07.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:07.565 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:42:07.565 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T22:42:07.565 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.3 2024-03-11T22:42:07.889 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:07 smithi073 systemd[1]: Stopping Ceph osd.3 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:08.220 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:07 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:07.881+0000 7f421c3be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:08.220 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:07 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:07.881+0000 7f421c3be700 -1 osd.3 155 *** Got signal Terminated *** 2024-03-11T22:42:08.220 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:07 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:07.881+0000 7f421c3be700 -1 osd.3 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:10.720 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:10 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:10.255+0000 7f42189b2700 -1 osd.3 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.409036+0000 front 2024-03-11T22:41:46.408823+0000 (oldest deadline 2024-03-11T22:42:09.908909+0000) 2024-03-11T22:42:10.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:10 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:10.627+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:10.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:10 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:10.595+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:11.470 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:11 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:11.210+0000 7f42189b2700 -1 osd.3 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.409036+0000 front 2024-03-11T22:41:46.408823+0000 (oldest deadline 2024-03-11T22:42:09.908909+0000) 2024-03-11T22:42:11.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:11.568+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:11.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:11.738+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:11.994 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:11 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:11.596+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:12.470 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:12 smithi073 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3[80510]: 2024-03-11T22:42:12.204+0000 7f42189b2700 -1 osd.3 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.409036+0000 front 2024-03-11T22:41:46.408823+0000 (oldest deadline 2024-03-11T22:42:09.908909+0000) 2024-03-11T22:42:12.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:12 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:12.738+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:12.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:12 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:12.561+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:12.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:12 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:12.551+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:13.493 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:13 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:13.228+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:13.564 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:13 smithi073 podman[113229]: 2024-03-11 22:42:13.293335238 +0000 UTC m=+5.509793117 container died 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:13.940 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:13 smithi073 podman[113229]: 2024-03-11 22:42:13.706537381 +0000 UTC m=+5.922995260 container cleanup 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T22:42:13.940 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:13 smithi073 bash[113229]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3 2024-03-11T22:42:13.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:13 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:13.576+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:13.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:13 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:13.517+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:13.994 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:13 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:13.703+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:14.220 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:13 smithi073 podman[113279]: 2024-03-11 22:42:13.933095835 +0000 UTC m=+0.640328732 container remove 810f2752eb26c61dff162c1d3ae98a1b29aba85175f8bdcf9db513940ab5ee29 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:42:14.493 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:14 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:14.232+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:14.634 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.232108887 +0000 UTC m=+0.017925923 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:14.635 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.375345305 +0000 UTC m=+0.161162328 container create 47f413d26734d7d8fe7214965ff18db116f193997a644f556a26d1252a4b69b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:42:14.635 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.552577611 +0000 UTC m=+0.338394648 container init 47f413d26734d7d8fe7214965ff18db116f193997a644f556a26d1252a4b69b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T22:42:14.635 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.55593492 +0000 UTC m=+0.341751943 container start 47f413d26734d7d8fe7214965ff18db116f193997a644f556a26d1252a4b69b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T22:42:14.635 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.627590396 +0000 UTC m=+0.413407457 container attach 47f413d26734d7d8fe7214965ff18db116f193997a644f556a26d1252a4b69b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T22:42:14.970 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 11 22:42:14 smithi073 podman[113370]: 2024-03-11 22:42:14.843206541 +0000 UTC m=+0.629023566 container died 47f413d26734d7d8fe7214965ff18db116f193997a644f556a26d1252a4b69b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:42:14.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:14 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:14.616+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:14.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:14 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:14.680+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:14.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:14 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:14.510+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:15.482 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T22:42:15.493 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:15.194+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:15.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:15.524 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T22:42:15.524 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T22:42:15.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.4 2024-03-11T22:42:15.900 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:15 smithi099 systemd[1]: Stopping Ceph osd.4 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:15.900 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:15.662+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:15.900 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:15.654+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:15.901 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:15.528+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:16.228 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:15.891+0000 7f577da29700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:16.229 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:15.891+0000 7f577da29700 -1 osd.4 155 *** Got signal Terminated *** 2024-03-11T22:42:16.229 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:15 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:15.891+0000 7f577da29700 -1 osd.4 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:16.229 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:16 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:16.220+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:16.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:16 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:16.707+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:16.994 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:16 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:16.668+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:16.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:16 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:16.535+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:17.688 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:17 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:17.500+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:17.688 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:17 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:17.263+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:17.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:17 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:17.743+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:17.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:17 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:17.679+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:18.679 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:18 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:18.526+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:18.679 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:18 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:18.282+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:18.993 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:18 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:18.727+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:18.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:18 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:18.670+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:19.650 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:19 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:19.517+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:19.650 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:19 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:19.325+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:19.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:19 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:19.641+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:19.994 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:19 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:19.722+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:19.994 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:19 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:19.722+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:20.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:20.503+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:20.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:20.604+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:20.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:20.604+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:20.744 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:20.358+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:47.387775+0000 front 2024-03-11T22:41:47.387742+0000 (oldest deadline 2024-03-11T22:42:13.188073+0000) 2024-03-11T22:42:20.744 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4[58145]: 2024-03-11T22:42:20.358+0000 7f577a01d700 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:54.289234+0000 front 2024-03-11T22:41:54.289294+0000 (oldest deadline 2024-03-11T22:42:19.589064+0000) 2024-03-11T22:42:21.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:20.755+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:21.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:20 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:20.755+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:21.513 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:21 smithi099 podman[78931]: 2024-03-11 22:42:21.250877786 +0000 UTC m=+5.535865312 container died 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T22:42:21.984 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:21.505+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:21.984 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:21 smithi099 podman[78931]: 2024-03-11 22:42:21.700524131 +0000 UTC m=+5.985511700 container cleanup 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T22:42:21.984 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:21 smithi099 bash[78931]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4 2024-03-11T22:42:21.985 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:21.747+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:21.985 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:21.747+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:21.985 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:21.651+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:21.985 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:21 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:21.651+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:22.243 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:21 smithi099 podman[78949]: 2024-03-11 22:42:21.977282534 +0000 UTC m=+0.731096118 container remove 97e1ed6508c36f559be0637a7cce0bc3dfad939b1fad614dbee383836e3e343a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T22:42:22.579 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:22.491+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:22.579 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:22.491+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:22.580 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.214383456 +0000 UTC m=+0.017985346 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:22.580 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.320065464 +0000 UTC m=+0.123667388 container create 61a0ebd3de0259f527e6d84ce7ca919178aae579e09626f2404b7252f4af5a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:22.580 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.57271355 +0000 UTC m=+0.376315445 container init 61a0ebd3de0259f527e6d84ce7ca919178aae579e09626f2404b7252f4af5a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T22:42:22.874 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.576051934 +0000 UTC m=+0.379653826 container start 61a0ebd3de0259f527e6d84ce7ca919178aae579e09626f2404b7252f4af5a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:42:22.874 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.656017029 +0000 UTC m=+0.459618924 container attach 61a0ebd3de0259f527e6d84ce7ca919178aae579e09626f2404b7252f4af5a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T22:42:22.874 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:22.715+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:22.875 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:22.715+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:22.875 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:22.686+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:22.875 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:22 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:22.686+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:23.243 INFO:journalctl@ceph.osd.4.smithi099.stdout:Mar 11 22:42:22 smithi099 podman[79044]: 2024-03-11 22:42:22.867000528 +0000 UTC m=+0.670602422 container died 61a0ebd3de0259f527e6d84ce7ca919178aae579e09626f2404b7252f4af5a95 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:42:23.592 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:23.491+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:23.593 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:23.491+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:23.622 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T22:42:23.666 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:23.666 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T22:42:23.666 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T22:42:23.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.5 2024-03-11T22:42:23.861 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:23.701+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:23.861 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:23.701+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:23.861 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:23 smithi099 systemd[1]: Stopping Ceph osd.5 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:23.861 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:23.751+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:23.861 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:23 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:23.751+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:24.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:24.004+0000 7f2c447d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:24.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:24.004+0000 7f2c447d2700 -1 osd.5 155 *** Got signal Terminated *** 2024-03-11T22:42:24.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:24.004+0000 7f2c447d2700 -1 osd.5 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:24.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:24.446+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:24.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:24.446+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:24.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:24.668+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:24.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:24.668+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:25.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:24.754+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:25.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:24 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:24.754+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:25.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:25.446+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:25.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:25.446+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:25.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:25.651+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:25.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:25.651+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:26.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:25.768+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:26.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:25 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:25.768+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:26.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:26.474+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:26.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:26.474+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:26.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:26.627+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:26.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:26.627+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:27.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:26.746+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:27.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:26.746+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:27.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:26 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:26.746+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:04.351928+0000 front 2024-03-11T22:42:04.351946+0000 (oldest deadline 2024-03-11T22:42:26.051885+0000) 2024-03-11T22:42:27.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:27.463+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:27.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:27.463+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:27.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:27.463+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:27.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:27.579+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:27.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:27.579+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:28.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:27.765+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:28.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:27.765+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:28.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:27 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:27.765+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:04.351928+0000 front 2024-03-11T22:42:04.351946+0000 (oldest deadline 2024-03-11T22:42:26.051885+0000) 2024-03-11T22:42:28.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:28.594+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:28.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:28.594+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:28.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:28.460+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:28.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:28.460+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:28.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:28.460+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:29.243 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:28.791+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.949413+0000 front 2024-03-11T22:41:46.949464+0000 (oldest deadline 2024-03-11T22:42:11.049221+0000) 2024-03-11T22:42:29.244 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:28.791+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.751109+0000 front 2024-03-11T22:41:56.751055+0000 (oldest deadline 2024-03-11T22:42:19.051041+0000) 2024-03-11T22:42:29.244 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:28 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5[60400]: 2024-03-11T22:42:28.791+0000 7f2c3cdbe700 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:04.351928+0000 front 2024-03-11T22:42:04.351946+0000 (oldest deadline 2024-03-11T22:42:26.051885+0000) 2024-03-11T22:42:29.743 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:29 smithi099 podman[79152]: 2024-03-11 22:42:29.60880607 +0000 UTC m=+5.720484836 container died 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T22:42:29.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:29 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:29.547+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:29.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:29 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:29.547+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:29.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:29 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:29.470+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:29.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:29 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:29.470+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:29.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:29 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:29.470+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:30.422 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:30 smithi099 podman[79152]: 2024-03-11 22:42:30.236138293 +0000 UTC m=+6.347817048 container cleanup 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T22:42:30.422 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:30 smithi099 bash[79152]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5 2024-03-11T22:42:30.743 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:30 smithi099 podman[79213]: 2024-03-11 22:42:30.414997478 +0000 UTC m=+0.806566562 container remove 33ae2416752054d6e2a0ff8aa301732998f75b06c3d28e3d04abaf17379b3e9c (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:42:30.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:30 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:30.575+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:30.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:30 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:30.575+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:30.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:30 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:30.512+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:30.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:30 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:30.512+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:30.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:30 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:30.512+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:31.176 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:30 smithi099 podman[79304]: 2024-03-11 22:42:30.747310668 +0000 UTC m=+0.017601632 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:31.176 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:30 smithi099 podman[79304]: 2024-03-11 22:42:30.857847947 +0000 UTC m=+0.128138920 container create 299073bb13d86145c5bab7bce70362c3f3a4d13fbf3935c8e1d2eeb31e50cd97 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T22:42:31.176 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:31 smithi099 podman[79304]: 2024-03-11 22:42:31.085670487 +0000 UTC m=+0.355961466 container init 299073bb13d86145c5bab7bce70362c3f3a4d13fbf3935c8e1d2eeb31e50cd97 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:31.176 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:31 smithi099 podman[79304]: 2024-03-11 22:42:31.088796422 +0000 UTC m=+0.359087397 container start 299073bb13d86145c5bab7bce70362c3f3a4d13fbf3935c8e1d2eeb31e50cd97 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:31.176 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:31 smithi099 podman[79304]: 2024-03-11 22:42:31.168994261 +0000 UTC m=+0.439285260 container attach 299073bb13d86145c5bab7bce70362c3f3a4d13fbf3935c8e1d2eeb31e50cd97 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T22:42:31.493 INFO:journalctl@ceph.osd.5.smithi099.stdout:Mar 11 22:42:31 smithi099 podman[79304]: 2024-03-11 22:42:31.375708455 +0000 UTC m=+0.645999430 container died 299073bb13d86145c5bab7bce70362c3f3a4d13fbf3935c8e1d2eeb31e50cd97 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T22:42:31.994 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:31.538+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:31.994 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:31.538+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:31.994 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:31.538+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:31.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:31.538+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:31.995 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:31.538+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:31.995 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:31 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:31.538+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:32.022 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T22:42:32.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:32.068 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T22:42:32.068 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T22:42:32.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.6 2024-03-11T22:42:32.374 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 systemd[1]: Stopping Ceph osd.6 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:32.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:32.509+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:32.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:32.509+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:32.509+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.366+0000 7f7c60331700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.366+0000 7f7c60331700 -1 osd.6 155 *** Got signal Terminated *** 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.366+0000 7f7c60331700 -1 osd.6 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.505+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.505+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:32.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:32 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:32.505+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:33.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:33.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:33.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:33.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:33.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:33.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:33.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:33.525+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:33.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:33.525+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:33.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:33.525+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:33.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:33 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:33.525+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:34.579 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:34.431+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:34.579 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:34.431+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:34.580 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:34.431+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:34.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:34.571+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:34.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:34.571+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:34.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:34.571+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:34.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:34 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:34.571+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:35.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:35.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:35.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:35.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:35.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:35.471+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:35.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:35.569+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:35.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:35.569+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:35.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:35.569+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:35.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:35 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:35.569+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:36.743 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:36.434+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:46.395007+0000 front 2024-03-11T22:41:46.395121+0000 (oldest deadline 2024-03-11T22:42:09.894881+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:36.434+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:55.696409+0000 front 2024-03-11T22:41:55.696309+0000 (oldest deadline 2024-03-11T22:42:20.396222+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6[62673]: 2024-03-11T22:42:36.434+0000 7f7c5891d700 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:00.396822+0000 front 2024-03-11T22:42:00.396794+0000 (oldest deadline 2024-03-11T22:42:30.997225+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:36.551+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:36.551+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:36.551+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:36.744 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:36 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:36.551+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:37.920 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:37 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:37.599+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:37.920 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:37 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:37.599+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:37.920 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:37 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:37.599+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:37.921 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:37 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:37.599+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:38.199 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:37 smithi099 podman[79410]: 2024-03-11 22:42:37.913203698 +0000 UTC m=+5.667166638 container died e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T22:42:38.493 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:38 smithi099 podman[79410]: 2024-03-11 22:42:38.366877088 +0000 UTC m=+6.120840023 container cleanup e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T22:42:38.493 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:38 smithi099 bash[79410]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6 2024-03-11T22:42:38.993 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:38 smithi099 podman[79428]: 2024-03-11 22:42:38.702012416 +0000 UTC m=+0.789119488 container remove e7b52a9fb49821de458066e577beeec5617e312dbedb3d00670d959a93e3cf14 (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:42:38.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:38 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:38.607+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:38.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:38 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:38.607+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:38.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:38 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:38.607+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:38.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:38 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:38.607+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:39.362 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:38.925176163 +0000 UTC m=+0.017441488 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:39.363 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:39.036201768 +0000 UTC m=+0.128467092 container create 235589e3d30134a328a1fd57edab98ad0087eaedc961a5045d2417dae3aae81b (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:42:39.363 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:39.263752799 +0000 UTC m=+0.356018139 container init 235589e3d30134a328a1fd57edab98ad0087eaedc961a5045d2417dae3aae81b (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T22:42:39.363 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:39.266782052 +0000 UTC m=+0.359047378 container start 235589e3d30134a328a1fd57edab98ad0087eaedc961a5045d2417dae3aae81b (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:42:39.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:39.355457061 +0000 UTC m=+0.447722397 container attach 235589e3d30134a328a1fd57edab98ad0087eaedc961a5045d2417dae3aae81b (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T22:42:39.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Mar 11 22:42:39 smithi099 podman[79517]: 2024-03-11 22:42:39.555648005 +0000 UTC m=+0.647913322 container died 235589e3d30134a328a1fd57edab98ad0087eaedc961a5045d2417dae3aae81b (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T22:42:39.941 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:39.611+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:39.941 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:39.611+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:39.941 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:39.611+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:39.941 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:39 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:39.611+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:40.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T22:42:40.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:40.285 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T22:42:40.285 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T22:42:40.286 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.7 2024-03-11T22:42:40.493 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 systemd[1]: Stopping Ceph osd.7 for f2f121a8-dff4-11ee-95c8-87774f69a715... 2024-03-11T22:42:40.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.595+0000 7f79ded88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:42:40.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.595+0000 7f79ded88700 -1 osd.7 155 *** Got signal Terminated *** 2024-03-11T22:42:40.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.595+0000 7f79ded88700 -1 osd.7 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:42:40.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.614+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:40.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.614+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:40.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.615+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:40.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:40 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:40.615+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:41.910 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:41 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:41.632+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:41.910 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:41 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:41.632+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:41.910 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:41 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:41.632+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:41.910 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:41 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:41.632+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:42.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:42.655+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:42.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:42.655+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:42.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:42.655+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:42.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:42 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:42.655+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:43.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:43.638+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:43.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:43.638+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:43.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:43.638+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:43.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:43.638+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:43.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:43 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:43.638+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.99:6804 osd.4 since back 2024-03-11T22:42:19.243320+0000 front 2024-03-11T22:42:19.243387+0000 (oldest deadline 2024-03-11T22:42:42.743331+0000) 2024-03-11T22:42:44.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:44.648+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-11T22:41:44.540514+0000 front 2024-03-11T22:41:44.540667+0000 (oldest deadline 2024-03-11T22:42:09.840378+0000) 2024-03-11T22:42:44.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:44.648+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6814 osd.1 since back 2024-03-11T22:41:56.241692+0000 front 2024-03-11T22:41:56.241775+0000 (oldest deadline 2024-03-11T22:42:21.541584+0000) 2024-03-11T22:42:44.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:44.648+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6822 osd.2 since back 2024-03-11T22:42:01.542146+0000 front 2024-03-11T22:42:01.542036+0000 (oldest deadline 2024-03-11T22:42:27.442003+0000) 2024-03-11T22:42:44.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:44.648+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.73:6830 osd.3 since back 2024-03-11T22:42:07.442472+0000 front 2024-03-11T22:42:07.442418+0000 (oldest deadline 2024-03-11T22:42:33.342457+0000) 2024-03-11T22:42:44.994 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:44 smithi099 ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7[64934]: 2024-03-11T22:42:44.648+0000 7f79d7374700 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.99:6804 osd.4 since back 2024-03-11T22:42:19.243320+0000 front 2024-03-11T22:42:19.243387+0000 (oldest deadline 2024-03-11T22:42:42.743331+0000) 2024-03-11T22:42:46.295 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:46 smithi099 podman[79623]: 2024-03-11 22:42:46.032312368 +0000 UTC m=+5.536152400 container died f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-11T22:42:46.646 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:46 smithi099 podman[79623]: 2024-03-11 22:42:46.45358632 +0000 UTC m=+5.957426361 container cleanup f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:42:46.646 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:46 smithi099 bash[79623]: ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7 2024-03-11T22:42:46.993 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:46 smithi099 podman[79652]: 2024-03-11 22:42:46.638511206 +0000 UTC m=+0.605697217 container remove f3cd6f3bf5ea681275eb9368ebedf3610b2aba335f5f3fbb68fd9034ef1688fa (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T22:42:47.391 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:46.970537828 +0000 UTC m=+0.018001731 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:42:47.391 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:47.081252402 +0000 UTC m=+0.128716295 container create 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T22:42:47.391 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:47.300675201 +0000 UTC m=+0.348139109 container init 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T22:42:47.391 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:47.30377858 +0000 UTC m=+0.351242473 container start 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T22:42:47.391 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:47.38384237 +0000 UTC m=+0.431306265 container attach 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T22:42:47.743 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:47 smithi099 podman[79748]: 2024-03-11 22:42:47.582426525 +0000 UTC m=+0.629890420 container died 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:42:48.211 INFO:journalctl@ceph.osd.7.smithi099.stdout:Mar 11 22:42:48 smithi099 podman[79774]: 2024-03-11 22:42:48.187554133 +0000 UTC m=+0.598889521 container remove 8616681178ce5f17f8abdb037000a4b2c51da7602e49aecc53e456fe491af46a (image=quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f2f121a8-dff4-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T22:42:48.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f2f121a8-dff4-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T22:42:48.258 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:42:48.258 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T22:42:48.259 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:45:46.412 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:46:08.000 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:46:08.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:46:08.068 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:46:08.069 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi073/crash 2024-03-11T22:46:08.070 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:46:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:tar: /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:46:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:46:08.125 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi099/crash 2024-03-11T22:46:08.126 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:46:08.186 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:46:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:46:08.187 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:46:08.188 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:46:08.219 INFO:teuthology.orchestra.run.smithi073.stdout:2024-03-11T22:22:07.774761+0000 mon.smithi073 (mon.0) 507 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:46:08.219 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:46:08.219 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[SEC\]' /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:46:08.289 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]' /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:46:08.358 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[WRN\]' /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:46:08.428 INFO:teuthology.orchestra.run.smithi073.stdout:2024-03-11T22:22:07.774761+0000 mon.smithi073 (mon.0) 507 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:46:08.428 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:46:08.428 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:46:08.471 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:46:08.498 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:: No such file or directory 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi073.lmopce.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi073.lmopce.log: /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr: 89.3% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi073.lmopce.log.gz 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi073.pglkvm.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.3.log: /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi073.pglkvm.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log: 85.7% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:46:08.499 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:46:08.501 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log: 91.5% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:46:08.502 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi073.log 2024-03-11T22:46:08.507 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:: No such file or directory 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi099.log 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi099.kagazk.log 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.6.log: /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi099.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi099.rcoixq.log 2024-03-11T22:46:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi099.kagazk.log: /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi099.rcoixq.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mds.foofs.smithi099.kagazk.log.gz 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.5.log: 91.6% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:46:08.510 INFO:teuthology.orchestra.run.smithi099.stderr: 85.7%/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log: -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:46:08.511 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:46:08.514 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi073.log: 89.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:46:08.518 INFO:teuthology.orchestra.run.smithi073.stderr: 91.4% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:46:08.522 INFO:teuthology.orchestra.run.smithi099.stderr: 91.4% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:46:08.560 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi099.rcoixq.log.gz 2024-03-11T22:46:08.831 INFO:teuthology.orchestra.run.smithi099.stderr: 92.6% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi099.log.gz 2024-03-11T22:46:08.888 INFO:teuthology.orchestra.run.smithi073.stderr: 88.3% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mgr.smithi073.pglkvm.log.gz 2024-03-11T22:46:09.877 INFO:teuthology.orchestra.run.smithi073.stderr: 92.0% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-mon.smithi073.log.gz 2024-03-11T22:46:21.033 INFO:teuthology.orchestra.run.smithi073.stderr: 94.5% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:46:23.568 INFO:teuthology.orchestra.run.smithi099.stderr: 93.9% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T22:46:25.655 INFO:teuthology.orchestra.run.smithi099.stderr: 94.2% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T22:46:25.658 INFO:teuthology.orchestra.run.smithi073.stderr: 94.2% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:46:26.970 INFO:teuthology.orchestra.run.smithi073.stderr: 94.2% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:46:28.893 INFO:teuthology.orchestra.run.smithi099.stderr: 94.1% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T22:46:29.070 INFO:teuthology.orchestra.run.smithi073.stderr: 94.0% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T22:46:29.072 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-11T22:46:29.072 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m20.590s 2024-03-11T22:46:29.073 INFO:teuthology.orchestra.run.smithi073.stderr:user 1m7.762s 2024-03-11T22:46:29.073 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m2.687s 2024-03-11T22:46:34.281 INFO:teuthology.orchestra.run.smithi099.stderr: 94.0% -- replaced with /var/log/ceph/f2f121a8-dff4-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T22:46:34.283 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-11T22:46:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m25.798s 2024-03-11T22:46:34.284 INFO:teuthology.orchestra.run.smithi099.stderr:user 1m15.770s 2024-03-11T22:46:34.284 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m2.919s 2024-03-11T22:46:34.284 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:46:34.284 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi073/log 2024-03-11T22:46:34.285 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:46:44.017 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi099/log 2024-03-11T22:46:44.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:46:54.679 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:46:54.679 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 --force 2024-03-11T22:46:55.053 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f2f121a8-dff4-11ee-95c8-87774f69a715 --force 2024-03-11T22:46:55.438 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:46:55.438 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:46:55.453 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:46:55.468 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:46:55.469 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:46:55.481 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:46:55.481 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:46:55.496 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:46:55.510 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:46:55.512 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:46:55.512 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-03-11T22:46:55.513 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 +4604us[+4604us] +/- 122ms 2024-03-11T22:46:55.513 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -1307us[-1342us] +/- 60ms 2024-03-11T22:46:55.513 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -1038us[-1073us] +/- 69ms 2024-03-11T22:46:55.513 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:46:55.522 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +4520us[+4520us] +/- 122ms 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1376us[-1346us] +/- 60ms 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -1120us[-1120us] +/- 69ms 2024-03-11T22:46:55.528 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:46:55.529 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:46:55.539 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:46:55.540 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:46:55.549 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:46:55.635 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:46:55.714 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:46:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:46:55.745 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:46:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:46:55.776 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:46:55.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:46:55.796 INFO:teuthology.task.internal:Duration was 2108.042839 seconds 2024-03-11T22:46:55.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:46:55.806 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:46:55.806 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:46:55.809 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:46:55.849 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:46:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:46:56.275 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:46:56.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2024-03-11T22:46:56.275 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:46:56.318 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-03-11T22:46:56.319 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:46:56.343 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:46:56.344 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:46:56.361 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:46:56.412 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:46:56.412 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:46:56.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:47:09.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:47:09.998 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:47:09.998 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:47:10.025 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:47:10.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:47:10.070 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:47:10.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:47:10.095 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2024-03-11T22:47:10.127 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-03-11T22:47:10.149 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:47:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:47:10.177 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:47:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:47:10.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:47:10.217 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:47:10.218 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi073 2024-03-11T22:47:10.218 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:47:10.283 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985/remote/smithi099 2024-03-11T22:47:10.284 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:47:10.332 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:47:10.332 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:47:10.334 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:47:10.380 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:47:10.391 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:47:10.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:47:10.402 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:47:10.402 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:47:10.404 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:47:10.416 INFO:teuthology.orchestra.run.smithi073.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:47 /home/ubuntu/cephtest 2024-03-11T22:47:10.435 INFO:teuthology.orchestra.run.smithi099.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:47 /home/ubuntu/cephtest 2024-03-11T22:47:10.436 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:47:10.448 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:47:10.472 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:47:10.502 DEBUG:teuthology.nuke:shortname: smithi073 2024-03-11T22:47:10.502 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:47:10.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:01:53.865377', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuWNmKhPR/5LHxsOOVBDRBoJL4b7C7EW2Szf0NaC7gJjhoB2Ts3xlfV5fD3c4bByL89icpJDfD6bbQgjQ6tLyk='} 2024-03-11T22:47:10.529 DEBUG:teuthology.nuke:shortname: smithi099 2024-03-11T22:47:10.529 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:47:10.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591985', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:01:53.866952', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA60YmM6yAyWy2wrFD9iNyQnzN1QDkJ3w+ZnMysKY8OJdLSEEOIGwEX1EGRVyvHw9BEHp5HI5DJ22I7HtKvj3W8='} 2024-03-11T22:47:10.582 INFO:teuthology.orchestra.console.smithi073:Power off 2024-03-11T22:47:10.582 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:47:10.608 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2024-03-11T22:47:10.608 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:10.627 INFO:teuthology.orchestra.console.smithi099:Power off 2024-03-11T22:47:10.627 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:47:10.638 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-03-11T22:47:10.651 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-03-11T22:47:10.652 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:10.677 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-11T22:47:14.638 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:14.678 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:14.748 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-03-11T22:47:14.788 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-11T22:47:18.754 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:18.788 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:18.870 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-03-11T22:47:18.899 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-11T22:47:22.873 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:22.900 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:47:22.982 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2024-03-11T22:47:23.011 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-03-11T22:47:23.084 INFO:teuthology.orchestra.console.smithi073:Power off completed 2024-03-11T22:47:23.112 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-03-11T22:47:23.216 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2108.0428388118744 failure_reason: '"2024-03-11T22:22:07.774761+0000 mon.smithi073 (mon.0) 507 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:47:23.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:47:23.299 INFO:teuthology.run:FAIL