2024-04-27T01:58:02.845 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T01:58:02.845 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --name teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541 --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/orig.config.yaml 2024-04-27T01:58:02.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T01:58:02.989 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675541' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 4854 sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a sleep_before_teardown: 0 subset: 40/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEYNNF19zBoVS1kVFXKzKx3oHddUa7F+gm1LxTF1S3xseqpR6Yg+Bh5hKPEE4jRdxE4jzk2CIvKkJSIIVI7gHc= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU/fT/00CE2Kkjl5/ikJuKfY8klddqJtX8hdbl4FL8HUrO/4pHyw+xVuOYXLvAGr4fT58n7r28aEwZdSK7r3zc= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-27T01:58:02.989 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-27T01:58:02.990 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-27T01:58:02.990 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T01:58:02.992 INFO:teuthology.task.internal:Checking packages... 2024-04-27T01:58:03.021 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a' 2024-04-27T01:58:03.021 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T01:58:03.021 INFO:teuthology.packaging:ref: None 2024-04-27T01:58:03.021 INFO:teuthology.packaging:tag: None 2024-04-27T01:58:03.022 INFO:teuthology.packaging:branch: squid 2024-04-27T01:58:03.022 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:58:03.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-27T01:58:03.241 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447.gb22e2ebd 2024-04-27T01:58:03.243 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T01:58:03.248 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T01:58:03.249 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T01:58:03.258 INFO:teuthology.task.internal:Saving configuration 2024-04-27T01:58:03.272 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T01:58:03.278 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T01:58:03.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:52:54.950847', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEYNNF19zBoVS1kVFXKzKx3oHddUa7F+gm1LxTF1S3xseqpR6Yg+Bh5hKPEE4jRdxE4jzk2CIvKkJSIIVI7gHc='} 2024-04-27T01:58:03.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:52:54.951755', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU/fT/00CE2Kkjl5/ikJuKfY8klddqJtX8hdbl4FL8HUrO/4pHyw+xVuOYXLvAGr4fT58n7r28aEwZdSK7r3zc='} 2024-04-27T01:58:03.324 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T01:58:03.330 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-27T01:58:03.330 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-27T01:58:03.330 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T01:58:03.409 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9c79383f70>, signals=[15]) 2024-04-27T01:58:03.409 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T01:58:03.415 INFO:teuthology.task.internal:Opening connections... 2024-04-27T01:58:03.415 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T01:58:03.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:58:03.497 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T01:58:03.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T01:58:03.570 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T01:58:03.576 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-04-27T01:58:03.595 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-04-27T01:58:03.596 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="9" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="9" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el9" 2024-04-27T01:58:03.654 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:LOGO="fedora-logo-icon" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T01:58:03.655 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T01:58:03.655 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-04-27T01:58:03.677 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-04-27T01:58:03.694 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-04-27T01:58:03.694 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-04-27T01:58:03.751 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T01:58:03.752 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T01:58:03.752 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T01:58:03.752 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-04-27T01:58:03.776 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T01:58:03.786 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T01:58:03.792 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T01:58:03.792 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-04-27T01:58:03.794 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-04-27T01:58:03.811 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T01:58:03.816 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T01:58:03.816 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-04-27T01:58:03.852 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-04-27T01:58:03.961 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T01:58:04.009 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T01:58:04.024 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T01:58:04.024 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T01:58:04.025 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T01:58:04.025 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T01:58:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:58:04.042 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-04-27T01:58:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:58:04.044 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-04-27T01:58:04.100 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-437.el9.x86_64 2024-04-27T01:58:04.100 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-437.el9.x86_64 2024-04-27T01:58:04.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-04-27T01:58:04.102 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-437.el9.x86_64 2024-04-27T01:58:04.102 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.14.0-437.el9.x86_64 2024-04-27T01:58:04.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum install -y kernel 2024-04-27T01:58:06.334 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-27T01:58:08.328 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-04-27T01:58:10.156 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-27T01:58:11.626 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-04-27T01:58:14.742 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.2 MB 00:00 2024-04-27T01:58:16.335 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-04-27T01:58:16.666 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-27T01:58:17.719 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-04-27T01:58:18.284 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-27T01:58:19.104 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-27T01:58:23.467 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-27T01:58:24.338 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-27T01:58:25.348 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T01:58:25.348 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T01:58:25.440 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-27T01:58:25.441 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-04-27T01:58:25.441 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-27T01:58:25.559 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2024-04-27T01:58:26.098 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:58:23 AM UTC. 2024-04-27T01:58:26.226 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-27T01:58:26.244 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.244 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-04-27T01:58:26.244 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.244 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-04-27T01:58:26.244 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-04-27T01:58:26.245 INFO:teuthology.orchestra.run.smithi077.stderr:Operation aborted. 2024-04-27T01:58:26.297 INFO:teuthology.orchestra.run.smithi077.stdout:Is this ok [y/N]: 2024-04-27T01:58:26.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2024-04-27T01:58:26.310 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T01:58:26.310 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T01:58:26.401 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-04-27T01:58:26.402 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-04-27T01:58:26.402 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-04-27T01:58:26.524 DEBUG:teuthology.orchestra.run.smithi008:> echo no | sudo yum reinstall kernel || true 2024-04-27T01:58:26.789 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:58:23 AM UTC. 2024-04-27T01:58:26.903 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-04-27T01:58:26.904 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.904 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-04-27T01:58:26.904 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.904 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-04-27T01:58:26.905 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-04-27T01:58:27.048 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:58:24 AM UTC. 2024-04-27T01:58:27.145 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.5 MB/s | 255 kB 00:00 2024-04-27T01:58:27.145 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:58:27.145 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.0 MB/s | 255 kB 00:00 2024-04-27T01:58:27.163 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-04-27T01:58:27.173 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-04-27T01:58:27.182 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.182 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:Reinstalling: 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 255 k 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 0 2024-04-27T01:58:27.183 INFO:teuthology.orchestra.run.smithi008.stderr:Operation aborted. 2024-04-27T01:58:27.237 INFO:teuthology.orchestra.run.smithi008.stdout:Is this ok [y/N]: 2024-04-27T01:58:27.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum reinstall -y kernel || true 2024-04-27T01:58:27.361 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-04-27T01:58:27.361 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-04-27T01:58:27.366 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-04-27T01:58:27.366 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-04-27T01:58:27.494 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-04-27T01:58:27.554 INFO:teuthology.orchestra.run.smithi077.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:58:27.554 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:27.754 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 01:58:24 AM UTC. 2024-04-27T01:58:27.875 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:Reinstalling: 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:27.876 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 255 k 2024-04-27T01:58:27.877 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 0 2024-04-27T01:58:27.877 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-04-27T01:58:28.271 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:28.271 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalled: 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T01:58:28.982 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-04-27T01:58:29.073 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-437.el9.x86_64.rpm 274 kB/s | 255 kB 00:00 2024-04-27T01:58:29.073 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:58:29.073 INFO:teuthology.orchestra.run.smithi008.stdout:Total 213 kB/s | 255 kB 00:01 2024-04-27T01:58:29.097 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-04-27T01:58:29.147 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T01:58:29.193 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T01:58:29.194 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T01:58:29.194 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T01:58:29.194 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-27T01:58:29.194 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T01:58:29.231 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-04-27T01:58:29.247 DEBUG:teuthology.parallel:result is None 2024-04-27T01:58:29.290 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-04-27T01:58:29.290 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-04-27T01:58:29.295 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-04-27T01:58:29.295 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-04-27T01:58:29.417 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-04-27T01:58:29.507 INFO:teuthology.orchestra.run.smithi008.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:58:29.507 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:30.187 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:30.187 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T01:58:30.854 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T01:58:30.855 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:30.855 INFO:teuthology.orchestra.run.smithi008.stdout:Reinstalled: 2024-04-27T01:58:30.855 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T01:58:30.855 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T01:58:30.855 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-04-27T01:58:31.045 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T01:58:31.092 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T01:58:31.092 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T01:58:31.092 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T01:58:31.092 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-27T01:58:31.092 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T01:58:31.126 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-04-27T01:58:31.148 DEBUG:teuthology.parallel:result is None 2024-04-27T01:58:31.148 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T01:58:31.156 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T01:58:31.156 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T01:58:31.191 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T01:58:31.209 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T01:58:31.216 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T01:58:31.222 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T01:58:31.222 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T01:58:31.246 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T01:58:31.279 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T01:58:31.285 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T01:58:31.285 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T01:58:31.311 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T01:58:31.335 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:58:31.344 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:58:31.374 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:58:31.384 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T01:58:31.385 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T01:58:31.392 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T01:58:31.393 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T01:58:31.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T01:58:31.420 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T01:58:31.428 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T01:58:31.429 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T01:58:31.460 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T01:58:31.476 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:58:31.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:58:31.607 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:58:31.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:58:31.733 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T01:58:31.733 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T01:58:31.797 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:58:31.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T01:58:31.892 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:58:31.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T01:58:32.015 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T01:58:32.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T01:58:32.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-04-27T01:58:32.082 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-04-27T01:58:32.119 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:58:32.166 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:58:32.525 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T01:58:32.533 INFO:teuthology.task.internal:Starting timer... 2024-04-27T01:58:32.533 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T01:58:32.543 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T01:58:32.551 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-27T01:58:32.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2024-04-27T01:58:32.593 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: 2024-04-27T01:58:32.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-04-27T01:58:32.641 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-04-27T01:58:32.642 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T01:58:32.642 DEBUG:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2024-04-27T01:58:32.674 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2024-04-27T01:58:32.674 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-04-27T01:58:32.698 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-04-27T01:58:32.699 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2024-04-27T01:58:32.699 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:58:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:58:32.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:58:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:58:32.766 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T01:58:32.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2024-04-27T01:58:32.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-04-27T01:58:32.832 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T01:58:32.841 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T01:58:32.861 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T01:58:32.862 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T02:02:09.011 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2024-04-27T02:02:09.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-27T02:02:09.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T02:02:09.091 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-04-27T02:02:09.127 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-04-27T02:02:09.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T02:02:09.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T02:02:09.200 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-27T02:02:09.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T02:02:09.282 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T02:02:09.292 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T02:02:09.292 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T02:02:09.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T02:02:09.294 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T02:02:09.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T02:02:09.322 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T02:02:09.336 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T02:02:09.349 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T02:02:09.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2024-04-27T02:02:09.363 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T02:02:09.370 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2024-04-27T02:02:09.384 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T02:02:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-04-27T02:02:09.395 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-04-27T02:02:09.397 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T02:02:09.408 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T02:02:09.420 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T02:02:09.459 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.463 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.472 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-27T02:02:09.476 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T02:02:09.476 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-27T02:02:09.476 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.477 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.477 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.477 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:02:09.477 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-27T02:02:09.594 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T02:02:09.595 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-27T02:02:09.595 INFO:tasks.cephadm:Cluster fsid is 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:02:09.596 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-27T02:02:09.596 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-27T02:02:09.596 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi008': '172.21.15.8', 'mon.smithi077': '172.21.15.77'} 2024-04-27T02:02:09.596 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-27T02:02:09.597 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname $(hostname -s) 2024-04-27T02:02:09.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-04-27T02:02:09.647 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-27T02:02:09.647 DEBUG:teuthology.orchestra.run.smithi008:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.099 INFO:teuthology.orchestra.run.smithi008.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 27 02:02 /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.100 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.529 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 27 02:02 /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.530 DEBUG:teuthology.orchestra.run.smithi008:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.549 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T02:02:10.574 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-27T02:02:10.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-27T02:02:10.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-27T02:02:12.156 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T02:02:12.249 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T02:02:39.618 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: "repo_digests": [ 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-27T02:02:39.619 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-27T02:02:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-04-27T02:02:44.104 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-27T02:02:44.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph 2024-04-27T02:02:44.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-04-27T02:02:44.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /etc/ceph 2024-04-27T02:02:44.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-04-27T02:02:44.249 INFO:tasks.cephadm:Writing seed config... 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-27T02:02:44.250 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-27T02:02:44.251 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-27T02:02:44.251 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-27T02:02:44.251 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:02:44.251 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-27T02:02:44.282 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 27e8cf32-043a-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-27T02:02:44.283 DEBUG:teuthology.orchestra.run.smithi008:mon.smithi008> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service 2024-04-27T02:02:44.324 INFO:tasks.cephadm:Bootstrapping... 2024-04-27T02:02:44.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:02:44.528 INFO:teuthology.orchestra.run.smithi008.stderr:-------------------------------------------------------------------------------- 2024-04-27T02:02:44.528 INFO:teuthology.orchestra.run.smithi008.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '27e8cf32-043a-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.8', '--skip-admin-label'] 2024-04-27T02:02:44.552 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-27T02:02:44.555 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying podman|docker is present... 2024-04-27T02:02:44.584 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-27T02:02:44.585 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying lvm2 is present... 2024-04-27T02:02:44.586 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying time synchronization is in place... 2024-04-27T02:02:44.593 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T02:02:44.599 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:02:44.606 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-04-27T02:02:44.612 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-04-27T02:02:44.613 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-04-27T02:02:44.613 INFO:teuthology.orchestra.run.smithi008.stderr:Repeating the final host check... 2024-04-27T02:02:44.635 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: 4.9.4-dev 2024-04-27T02:02:44.637 INFO:teuthology.orchestra.run.smithi008.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-27T02:02:44.637 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl is present 2024-04-27T02:02:44.637 INFO:teuthology.orchestra.run.smithi008.stderr:lvcreate is present 2024-04-27T02:02:44.643 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T02:02:44.649 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:02:44.656 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: enabled 2024-04-27T02:02:44.662 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: active 2024-04-27T02:02:44.663 INFO:teuthology.orchestra.run.smithi008.stderr:Unit chronyd.service is enabled and running 2024-04-27T02:02:44.663 INFO:teuthology.orchestra.run.smithi008.stderr:Host looks OK 2024-04-27T02:02:44.663 INFO:teuthology.orchestra.run.smithi008.stderr:Cluster fsid: 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:02:44.663 INFO:teuthology.orchestra.run.smithi008.stderr:Acquiring lock 139702029112032 on /run/cephadm/27e8cf32-043a-11ef-bc93-c7b262605968.lock 2024-04-27T02:02:44.663 INFO:teuthology.orchestra.run.smithi008.stderr:Lock 139702029112032 acquired on /run/cephadm/27e8cf32-043a-11ef-bc93-c7b262605968.lock 2024-04-27T02:02:44.664 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 3300 ... 2024-04-27T02:02:44.664 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying IP 172.21.15.8 port 6789 ... 2024-04-27T02:02:44.664 INFO:teuthology.orchestra.run.smithi008.stderr:Base mon IP is 172.21.15.8, final addrv is [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-04-27T02:02:44.666 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.8 metric 100 2024-04-27T02:02:44.666 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.8 metric 100 2024-04-27T02:02:44.669 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-27T02:02:44.669 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-27T02:02:44.669 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:25fe/64 scope link noprefixroute 2024-04-27T02:02:44.672 INFO:teuthology.orchestra.run.smithi008.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-27T02:02:44.673 INFO:teuthology.orchestra.run.smithi008.stderr:Mon IP `172.21.15.8` is in CIDR network `172.21.0.0/20` 2024-04-27T02:02:44.673 INFO:teuthology.orchestra.run.smithi008.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-27T02:02:44.673 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T02:02:44.780 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-27T02:02:45.526 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Getting image source signatures 2024-04-27T02:02:45.526 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-27T02:02:45.526 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-27T02:02:45.527 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-27T02:02:45.527 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-27T02:02:45.527 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-27T02:02:45.527 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-27T02:02:45.688 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: Writing manifest to image destination 2024-04-27T02:02:45.840 INFO:teuthology.orchestra.run.smithi008.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-27T02:02:46.259 INFO:teuthology.orchestra.run.smithi008.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-27T02:02:46.748 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-27T02:02:46.749 INFO:teuthology.orchestra.run.smithi008.stderr:Extracting ceph user uid/gid from container image... 2024-04-27T02:02:47.114 INFO:teuthology.orchestra.run.smithi008.stderr:stat: 167 167 2024-04-27T02:02:47.620 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial keys... 2024-04-27T02:02:47.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQDHXCxmR8EkOBAAI15FI9iVbeP/LJ9AgAeKEQ== 2024-04-27T02:02:55.388 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQDOXCxmX9r0IRAAAbk0g7LTR2uDLlVPYP7uFA== 2024-04-27T02:02:58.470 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph-authtool: AQDSXCxmxupGGBAAEyXQomoJszA0l5otJld5LQ== 2024-04-27T02:02:58.985 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial monmap... 2024-04-27T02:02:59.734 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T02:02:59.735 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-27T02:02:59.735 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:02:59.735 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr:monmaptool for smithi008 [v2:172.21.15.8:3300,v1:172.21.15.8:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr:setting min_mon_release = octopus 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: set fsid to 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:03:00.241 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mon... 2024-04-27T02:03:01.106 INFO:teuthology.orchestra.run.smithi008.stderr:create mon.smithi008 on 2024-04-27T02:03:01.336 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-27T02:03:01.488 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target. 2024-04-27T02:03:01.488 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target. 2024-04-27T02:03:01.837 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Unit ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service not loaded. 2024-04-27T02:03:01.844 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target.wants/ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service → /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968@.service. 2024-04-27T02:03:02.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:02 smithi008 podman[21501]: 2024-04-27 02:03:02.481531575 +0000 UTC m=+0.329925061 container start 217c9d464a7c951f003c5fde7003e5fbe5d3db1fe223d36ed043694f25117cf9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, version=8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-27T02:03:02.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:02 smithi008 ceph-mon[21528]: mkfs 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:02.555 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-27T02:03:02.563 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:03:02.564 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-27T02:03:02.564 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-04-27T02:03:02.565 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon to start... 2024-04-27T02:03:02.565 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mon... 2024-04-27T02:03:02.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:02 smithi008 ceph-mon[21528]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-27T02:03:02.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:02 smithi008 bash[21501]: 217c9d464a7c951f003c5fde7003e5fbe5d3db1fe223d36ed043694f25117cf9 2024-04-27T02:03:02.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:02 smithi008 systemd[1]: Started Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:03:03.076 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: cluster: 2024-04-27T02:03:03.076 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: id: 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:03.076 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-27T02:03:03.076 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: services: 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi008 (age 0.576591s) 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: data: 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: pgs: 2024-04-27T02:03:03.077 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: fsmap 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: mgrmap e1: no daemons active 2024-04-27T02:03:03.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:03 smithi008 ceph-mon[21528]: from='client.? 172.21.15.8:0/4051035114' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-27T02:03:05.329 INFO:teuthology.orchestra.run.smithi008.stderr:mon is available 2024-04-27T02:03:05.329 INFO:teuthology.orchestra.run.smithi008.stderr:Assimilating anything we can from ceph.conf... 2024-04-27T02:03:05.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:05.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-04-27T02:03:05.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:05.839 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-27T02:03:05.840 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-27T02:03:06.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:05 smithi008 ceph-mon[21528]: from='client.? 172.21.15.8:0/981193126' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T02:03:06.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:05 smithi008 ceph-mon[21528]: from='client.? 172.21.15.8:0/981193126' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-27T02:03:06.359 INFO:teuthology.orchestra.run.smithi008.stderr:Generating new minimal ceph.conf... 2024-04-27T02:03:07.154 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:06 smithi008 ceph-mon[21528]: from='client.? 172.21.15.8:0/2909339656' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:03:07.450 INFO:teuthology.orchestra.run.smithi008.stderr:Restarting the monitor... 2024-04-27T02:03:07.892 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:07 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:03:07.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:07 smithi008 bash[21748]: Error: no container with name or ID "ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon.smithi008" found: no such container 2024-04-27T02:03:07.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:07 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[21519]: 2024-04-27T02:03:07.620+0000 7f9f07a8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:03:07.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:07 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[21519]: 2024-04-27T02:03:07.620+0000 7f9f07a8f700 -1 mon.smithi008@0(leader) e1 *** Got Signal Terminated *** 2024-04-27T02:03:07.893 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:07 smithi008 podman[21759]: 2024-04-27 02:03:07.66214142 +0000 UTC m=+0.129786081 container died 217c9d464a7c951f003c5fde7003e5fbe5d3db1fe223d36ed043694f25117cf9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, RELEASE=HEAD, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.expose-services=) 2024-04-27T02:03:08.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21759]: 2024-04-27 02:03:08.011821939 +0000 UTC m=+0.479466607 container cleanup 217c9d464a7c951f003c5fde7003e5fbe5d3db1fe223d36ed043694f25117cf9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, RELEASE=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:03:08.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 bash[21759]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008 2024-04-27T02:03:08.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21773]: 2024-04-27 02:03:08.170969193 +0000 UTC m=+0.501498892 container remove 217c9d464a7c951f003c5fde7003e5fbe5d3db1fe223d36ed043694f25117cf9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, architecture=x86_64, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, version=8, name=centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-27T02:03:08.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 bash[21787]: Error: no container with name or ID "ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon.smithi008" found: no such container 2024-04-27T02:03:08.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Deactivated successfully. 2024-04-27T02:03:08.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:03:08.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 systemd[1]: Starting Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:03:08.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21854]: 2024-04-27 02:03:08.449861675 +0000 UTC m=+0.099880421 container create 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, maintainer=Guillaume Abrioux , architecture=x86_64, name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-27T02:03:08.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21854]: 2024-04-27 02:03:08.369803844 +0000 UTC m=+0.019822600 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-27T02:03:08.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21854]: 2024-04-27 02:03:08.649029277 +0000 UTC m=+0.299048026 container init 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, maintainer=Guillaume Abrioux , name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, version=8, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public) 2024-04-27T02:03:08.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 podman[21854]: 2024-04-27 02:03:08.651924699 +0000 UTC m=+0.301943448 container start 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream, distribution-scope=public, maintainer=Guillaume Abrioux , version=8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-27T02:03:08.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: pidfile_write: ignore empty --pid-file 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: load: jerasure load: lrc 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: RocksDB version: 6.15.5 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Compile date Apr 18 2022 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: DB SUMMARY 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: DB Session ID: BJ66TF6HEH2HXNGVLHZ6 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: CURRENT file: CURRENT 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000008.sst 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000010.log size: 83178 ; 2024-04-27T02:03:08.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.error_if_exists: 0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.create_if_missing: 0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.paranoid_checks: 1 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.env: 0x55fb974c6860 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.fs: Posix File System 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.info_log: 0x55fb997b3fa0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.statistics: (nil) 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.use_fsync: 0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_log_file_size: 0 2024-04-27T02:03:08.706 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_fallocate: 1 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.use_direct_reads: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T02:03:08.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.db_log_dir: 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi008/store.db 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.write_buffer_manager: 0x55fb998a4330 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-27T02:03:08.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.unordered_write: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.row_cache: None 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.wal_filter: None 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T02:03:08.709 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.preserve_deletes: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.two_write_queues: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.atomic_flush: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.log_readahead_size: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T02:03:08.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_background_jobs: 2 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_background_compactions: -1 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_subcompactions: 1 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_open_files: -1 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T02:03:08.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_background_flushes: -1 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Compression algorithms supported: 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kZSTD supported: 0 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kXpressCompression supported: 0 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kLZ4Compression supported: 1 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kBZip2Compression supported: 0 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kZlibCompression supported: 1 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: kSnappyCompression supported: 1 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T02:03:08.712 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.merge_operator: 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_filter: None 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fb9977fd00) 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-04-27T02:03:08.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: hash_index_allow_collision: 1 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 1 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x55fb997ea170 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-04-27T02:03:08.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 4 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T02:03:08.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T02:03:08.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression: NoCompression 2024-04-27T02:03:08.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T02:03:08.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T02:03:08.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T02:03:08.716 INFO:teuthology.orchestra.run.smithi008.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.num_levels: 7 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T02:03:08.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T02:03:08.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T02:03:08.719 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.arena_block_size: 4194304 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T02:03:08.720 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.table_properties_collectors: 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.inplace_update_support: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.bloom_locality: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.max_successive_merges: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.ttl: 2592000 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.enable_blob_files: false 2024-04-27T02:03:08.721 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.min_blob_size: 0 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183388677609, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183388678440, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79616, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77880, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9900, "raw_average_key_size": 47, "raw_value_size": 72176, "raw_average_value_size": 343, "num_data_blocks": 10, "num_entries": 210, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714183388, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c1fe1a8e-8e48-45f9-8afd-9e2e357c9aad", "db_session_id": "BJ66TF6HEH2HXNGVLHZ6"}} 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-27T02:03:08.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714183388678772, "job": 1, "event": "recovery_finished"} 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55fb997d0a80 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: DB pointer 0x55fb997e0000 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:03:08.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T02:03:08.724 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 24.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 24.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 2/0 79.55 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 79.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T02:03:08.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.6 0.00 0.00 1 0.001 0 0 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 24.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???) e1 preinit fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:08.726 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).mds e1 new map 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).mds e1 print_map 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e1 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: -1 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: No filesystems configured 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 bash[21854]: 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f 2024-04-27T02:03:08.727 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 systemd[1]: Started Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:03:09.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mon.smithi008 is new leader, mons smithi008 in quorum (ranks 0) 2024-04-27T02:03:09.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: monmap e1: 1 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} 2024-04-27T02:03:09.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: fsmap 2024-04-27T02:03:09.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T02:03:09.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:08 smithi008 ceph-mon[21873]: mgrmap e1: no daemons active 2024-04-27T02:03:10.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:10 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/17548763' entity='client.admin' 2024-04-27T02:03:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-27T02:03:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:03:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:Creating mgr... 2024-04-27T02:03:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:Verifying port 9283 ... 2024-04-27T02:03:10.783 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Failed to reset failed state of unit ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mgr.smithi008.exkhsc.service: Unit ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mgr.smithi008.exkhsc.service not loaded. 2024-04-27T02:03:10.790 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: Created symlink /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968.target.wants/ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mgr.smithi008.exkhsc.service → /etc/systemd/system/ceph-27e8cf32-043a-11ef-bc93-c7b262605968@.service. 2024-04-27T02:03:11.435 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-27T02:03:11.442 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:03:11.442 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-27T02:03:11.443 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to enable service . firewalld.service is not available 2024-04-27T02:03:11.451 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-27T02:03:11.458 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:03:11.459 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-27T02:03:11.459 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-27T02:03:11.459 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr to start... 2024-04-27T02:03:11.459 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr... 2024-04-27T02:03:11.965 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "27e8cf32-043a-11ef-bc93-c7b262605968", 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:11.966 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T02:03:11.967 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T02:03:11.971 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-27T02:03:11.972 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-27T02:03:02.519046+0000", 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T02:03:11.973 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:12.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:12 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1421027270' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T02:03:12.498 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (1/15)... 2024-04-27T02:03:14.992 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "27e8cf32-043a-11ef-bc93-c7b262605968", 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-27T02:03:14.993 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T02:03:14.994 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-27T02:03:14.995 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-27T02:03:02.519046+0000", 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T02:03:14.996 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:15.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:15 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3101864264' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T02:03:15.504 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (2/15)... 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: mgrmap e2: smithi008.exkhsc(active, starting, since 0.00292417s) 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:03:17.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:03:17.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:03:17.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:17.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:17.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:17 smithi008 ceph-mon[21873]: from='mgr.14100 172.21.15.8:0/2920274565' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:18.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:18.016 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "27e8cf32-043a-11ef-bc93-c7b262605968", 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T02:03:18.017 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T02:03:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T02:03:18.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": false, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T02:03:18.023 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-27T02:03:02.519046+0000", 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T02:03:18.024 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:18.536 INFO:teuthology.orchestra.run.smithi008.stderr:mgr not available, waiting (3/15)... 2024-04-27T02:03:18.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:18 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/665905914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T02:03:18.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:18 smithi008 ceph-mon[21873]: mgrmap e3: smithi008.exkhsc(active, since 1.00634s) 2024-04-27T02:03:20.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:20 smithi008 ceph-mon[21873]: mgrmap e4: smithi008.exkhsc(active, since 2s) 2024-04-27T02:03:21.216 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsid": "27e8cf32-043a-11ef-bc93-c7b262605968", 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "health": { 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 0 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "smithi008" 2024-04-27T02:03:21.217 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T02:03:21.218 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T02:03:21.222 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "iostat", 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "nfs", 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "restful" 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ], 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:21.223 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "modified": "2024-04-27T02:03:02.519046+0000", 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "services": {} 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: }, 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T02:03:21.224 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:21.834 INFO:teuthology.orchestra.run.smithi008.stderr:mgr is available 2024-04-27T02:03:21.958 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:21 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1074099834' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [global] 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: fsid = 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-27T02:03:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [mgr] 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: [osd] 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-27T02:03:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-27T02:03:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:22 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3245584913' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T02:03:23.184 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling cephadm module... 2024-04-27T02:03:24.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:24 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/795833930' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-27T02:03:25.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:25 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/795833930' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-27T02:03:25.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:25 smithi008 ceph-mon[21873]: mgrmap e5: smithi008.exkhsc(active, since 7s) 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.exkhsc", 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-27T02:03:25.853 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:26.378 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-04-27T02:03:26.378 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 5... 2024-04-27T02:03:26.770 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:26 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2042201828' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T02:03:32.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:32.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-27T02:03:32.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-04-27T02:03:32.022 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: Active manager daemon smithi008.exkhsc restarted 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: osdmap e2: 0 total, 0 up, 0 in 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: mgrmap e6: smithi008.exkhsc(active, starting, since 0.0527172s) 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:03:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:32.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:31 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:32.559 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 5 is available 2024-04-27T02:03:32.560 INFO:teuthology.orchestra.run.smithi008.stderr:Setting orchestrator backend to cephadm... 2024-04-27T02:03:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: [27/Apr/2024:02:03:31] ENGINE Bus STARTING 2024-04-27T02:03:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: [27/Apr/2024:02:03:31] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:03:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: [27/Apr/2024:02:03:31] ENGINE Bus STARTED 2024-04-27T02:03:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T02:03:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: mgrmap e7: smithi008.exkhsc(active, since 1.05536s) 2024-04-27T02:03:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:33 smithi008 ceph-mon[21873]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T02:03:34.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:34 smithi008 ceph-mon[21873]: mgrmap e8: smithi008.exkhsc(active, since 2s) 2024-04-27T02:03:34.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:34 smithi008 ceph-mon[21873]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:34.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:34 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:34.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:34 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:34.275 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: value unchanged 2024-04-27T02:03:34.755 INFO:teuthology.orchestra.run.smithi008.stderr:Generating ssh key... 2024-04-27T02:03:35.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:35 smithi008 ceph-mon[21873]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:36.581 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHhVyYxV3l2pnC7iqSetsBrd6Y+Um/kquOHukfHiNNvR23O6Qfk4N+44RpInXWdDSHU4BIPhVsRmbkf+eMbHOqEE02nt5addulbJ8UoPQhyFYUsXXDSgRtXKF8p4K2DLleIgEJId5IsGWmGApFaf27bdmq9F+Z2PME3n+L8Oru2nFy/wg1+HDNc8m2Kw6LnSO8Yfkr+qezByZEo4f0mI9O8aBqzitD+a1CvBJfKttQNXQCUoJ4ckb+BAJGr2CusTXkgdCEEK9s7G6OmQqGhUrY0LRXTYI9/i8LabRza9xehRPAq/ZdqkKQXMo2UL6QyZZfITKdCuZePqPl/NRchc6AIH7VYII1yZWl5ksTttgc7BJpa3FdBYOuO7nIfMgIEgB76b5QsmEkVp4AZ7KlBv7lvelkLa8pp2UHt+HQHKx00Lm9tZ64evQtxv5Tkm0J9glkmBfEmQdvDA2E7FoX+nW/2ZgIGPgYLkrcgyej+qMBarzdDRZvQtjgqZtMhdKH7BU= ceph-27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:03:36.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:36 smithi008 ceph-mon[21873]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:36.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:36 smithi008 ceph-mon[21873]: Generating ssh key... 2024-04-27T02:03:36.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:36 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:36.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:36 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:37.136 INFO:teuthology.orchestra.run.smithi008.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-27T02:03:37.136 INFO:teuthology.orchestra.run.smithi008.stderr:Adding key to root@localhost authorized_keys... 2024-04-27T02:03:37.136 INFO:teuthology.orchestra.run.smithi008.stderr:Adding host smithi008... 2024-04-27T02:03:37.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:37 smithi008 ceph-mon[21873]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:38.609 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Added host 'smithi008' with addr '172.21.15.8' 2024-04-27T02:03:39.200 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mon service with default placement... 2024-04-27T02:03:39.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:39 smithi008 ceph-mon[21873]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi008", "addr": "172.21.15.8", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:39.801 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:39 smithi008 ceph-mon[21873]: Deploying cephadm binary to smithi008 2024-04-27T02:03:39.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:39 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:39.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:39 smithi008 ceph-mon[21873]: Added host smithi008 2024-04-27T02:03:39.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:39 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:03:39.802 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-27T02:03:40.500 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying mgr service with default placement... 2024-04-27T02:03:41.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:40 smithi008 ceph-mon[21873]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:41.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:40 smithi008 ceph-mon[21873]: Saving service mon spec with placement count:5 2024-04-27T02:03:41.069 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:40 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:41.191 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-27T02:03:42.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:42 smithi008 ceph-mon[21873]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:42.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:42 smithi008 ceph-mon[21873]: Saving service mgr spec with placement count:2 2024-04-27T02:03:42.221 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:42 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:42.653 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying crash service with default placement... 2024-04-27T02:03:43.260 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-27T02:03:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:43 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:43.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:43 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:43.911 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying prometheus service with default placement... 2024-04-27T02:03:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:44 smithi008 ceph-mon[21873]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:44 smithi008 ceph-mon[21873]: Saving service crash spec with placement * 2024-04-27T02:03:44.533 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:44 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:44.974 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-27T02:03:45.629 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying grafana service with default placement... 2024-04-27T02:03:45.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:45 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:45.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:45 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:45.849 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:45 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:46.255 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-27T02:03:46.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:46 smithi008 ceph-mon[21873]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:46.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:46 smithi008 ceph-mon[21873]: Saving service prometheus spec with placement count:1 2024-04-27T02:03:46.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:46 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:47.280 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying node-exporter service with default placement... 2024-04-27T02:03:47.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:47 smithi008 ceph-mon[21873]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:47.850 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:47 smithi008 ceph-mon[21873]: Saving service grafana spec with placement count:1 2024-04-27T02:03:48.241 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-27T02:03:48.847 INFO:teuthology.orchestra.run.smithi008.stderr:Deploying alertmanager service with default placement... 2024-04-27T02:03:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:49 smithi008 ceph-mon[21873]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:49 smithi008 ceph-mon[21873]: Saving service node-exporter spec with placement * 2024-04-27T02:03:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:49 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:49.495 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-27T02:03:50.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:50 smithi008 ceph-mon[21873]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:03:50.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:50 smithi008 ceph-mon[21873]: Saving service alertmanager spec with placement count:1 2024-04-27T02:03:50.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:50 smithi008 ceph-mon[21873]: from='mgr.14120 172.21.15.8:0/2641328949' entity='mgr.smithi008.exkhsc' 2024-04-27T02:03:51.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:51 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2312547462' entity='client.admin' 2024-04-27T02:03:52.475 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling the dashboard module... 2024-04-27T02:03:52.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:52 smithi008 ceph-mon[21873]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:03:52.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:52 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/179881258' entity='client.admin' 2024-04-27T02:03:53.981 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:53 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3528033150' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-27T02:03:54.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:54 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3528033150' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-27T02:03:54.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:54 smithi008 ceph-mon[21873]: mgrmap e9: smithi008.exkhsc(active, since 22s) 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "available": true, 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "active_name": "smithi008.exkhsc", 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-27T02:03:55.058 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:03:55.812 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:03:55 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2933835417' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T02:03:55.814 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for the mgr to restart... 2024-04-27T02:03:55.814 INFO:teuthology.orchestra.run.smithi008.stderr:Waiting for mgr epoch 9... 2024-04-27T02:04:01.031 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: { 2024-04-27T02:04:01.031 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-27T02:04:01.032 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: "initialized": true 2024-04-27T02:04:01.032 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: } 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: Active manager daemon smithi008.exkhsc restarted 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: osdmap e3: 0 total, 0 up, 0 in 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: mgrmap e10: smithi008.exkhsc(active, starting, since 0.0525714s) 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:01.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:04:01.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:04:01.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:00 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:01.684 INFO:teuthology.orchestra.run.smithi008.stderr:mgr epoch 9 is available 2024-04-27T02:04:01.684 INFO:teuthology.orchestra.run.smithi008.stderr:Generating a dashboard self-signed certificate... 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:00] ENGINE Bus STARTING 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:00] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:00] ENGINE Bus STARTED 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: mgrmap e11: smithi008.exkhsc(active, since 1.0548s) 2024-04-27T02:04:02.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:02 smithi008 ceph-mon[21873]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T02:04:02.374 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-27T02:04:02.974 INFO:teuthology.orchestra.run.smithi008.stderr:Creating initial admin user... 2024-04-27T02:04:03.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:03 smithi008 ceph-mon[21873]: mgrmap e12: smithi008.exkhsc(active, since 2s) 2024-04-27T02:04:03.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:03 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:03.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:03 smithi008 ceph-mon[21873]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:03.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:03 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:03.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:03 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:03.850 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$k2vsgChlJy8lYhNuSpuerOUxChQRid1wcwBD1eive2V3wR3rCDJu6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714183443, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-27T02:04:04.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:04 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:04.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:04 smithi008 ceph-mon[21873]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:04.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:04 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:05.193 INFO:teuthology.orchestra.run.smithi008.stderr:Fetching dashboard port number... 2024-04-27T02:04:05.873 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: 8443 2024-04-27T02:04:06.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:05 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3915506246' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-27T02:04:06.589 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: disabled 2024-04-27T02:04:06.598 INFO:teuthology.orchestra.run.smithi008.stderr:systemctl: inactive 2024-04-27T02:04:06.598 INFO:teuthology.orchestra.run.smithi008.stderr:firewalld.service is not enabled 2024-04-27T02:04:06.598 INFO:teuthology.orchestra.run.smithi008.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-27T02:04:06.600 INFO:teuthology.orchestra.run.smithi008.stderr:Ceph Dashboard is now available at: 2024-04-27T02:04:06.600 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:06.600 INFO:teuthology.orchestra.run.smithi008.stderr: URL: https://smithi008:8443/ 2024-04-27T02:04:06.600 INFO:teuthology.orchestra.run.smithi008.stderr: User: admin 2024-04-27T02:04:06.601 INFO:teuthology.orchestra.run.smithi008.stderr: Password: g4rt10k0dt 2024-04-27T02:04:06.601 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:06.601 INFO:teuthology.orchestra.run.smithi008.stderr:Enabling autotune for osd_memory_target 2024-04-27T02:04:08.130 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:07 smithi008 ceph-mon[21873]: mgrmap e13: smithi008.exkhsc(active, since 6s) 2024-04-27T02:04:08.521 INFO:teuthology.orchestra.run.smithi008.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-27T02:04:09.113 INFO:teuthology.orchestra.run.smithi008.stderr:You can access the Ceph CLI with: 2024-04-27T02:04:09.113 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.113 INFO:teuthology.orchestra.run.smithi008.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:04:09.113 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: ceph telemetry on 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr:For more information see: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:04:09.114 INFO:teuthology.orchestra.run.smithi008.stderr:Bootstrap complete. 2024-04-27T02:04:09.154 INFO:tasks.cephadm:Fetching config... 2024-04-27T02:04:09.154 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:04:09.154 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-27T02:04:09.174 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-27T02:04:09.174 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:04:09.174 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-27T02:04:09.229 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-27T02:04:09.230 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:04:09.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/keyring of=/dev/stdout 2024-04-27T02:04:09.294 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-27T02:04:09.295 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:04:09.295 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-27T02:04:09.350 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-27T02:04:09.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHhVyYxV3l2pnC7iqSetsBrd6Y+Um/kquOHukfHiNNvR23O6Qfk4N+44RpInXWdDSHU4BIPhVsRmbkf+eMbHOqEE02nt5addulbJ8UoPQhyFYUsXXDSgRtXKF8p4K2DLleIgEJId5IsGWmGApFaf27bdmq9F+Z2PME3n+L8Oru2nFy/wg1+HDNc8m2Kw6LnSO8Yfkr+qezByZEo4f0mI9O8aBqzitD+a1CvBJfKttQNXQCUoJ4ckb+BAJGr2CusTXkgdCEEK9s7G6OmQqGhUrY0LRXTYI9/i8LabRza9xehRPAq/ZdqkKQXMo2UL6QyZZfITKdCuZePqPl/NRchc6AIH7VYII1yZWl5ksTttgc7BJpa3FdBYOuO7nIfMgIEgB76b5QsmEkVp4AZ7KlBv7lvelkLa8pp2UHt+HQHKx00Lm9tZ64evQtxv5Tkm0J9glkmBfEmQdvDA2E7FoX+nW/2ZgIGPgYLkrcgyej+qMBarzdDRZvQtjgqZtMhdKH7BU= ceph-27e8cf32-043a-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T02:04:09.442 INFO:teuthology.orchestra.run.smithi008.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHhVyYxV3l2pnC7iqSetsBrd6Y+Um/kquOHukfHiNNvR23O6Qfk4N+44RpInXWdDSHU4BIPhVsRmbkf+eMbHOqEE02nt5addulbJ8UoPQhyFYUsXXDSgRtXKF8p4K2DLleIgEJId5IsGWmGApFaf27bdmq9F+Z2PME3n+L8Oru2nFy/wg1+HDNc8m2Kw6LnSO8Yfkr+qezByZEo4f0mI9O8aBqzitD+a1CvBJfKttQNXQCUoJ4ckb+BAJGr2CusTXkgdCEEK9s7G6OmQqGhUrY0LRXTYI9/i8LabRza9xehRPAq/ZdqkKQXMo2UL6QyZZfITKdCuZePqPl/NRchc6AIH7VYII1yZWl5ksTttgc7BJpa3FdBYOuO7nIfMgIEgB76b5QsmEkVp4AZ7KlBv7lvelkLa8pp2UHt+HQHKx00Lm9tZ64evQtxv5Tkm0J9glkmBfEmQdvDA2E7FoX+nW/2ZgIGPgYLkrcgyej+qMBarzdDRZvQtjgqZtMhdKH7BU= ceph-27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:04:09.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHhVyYxV3l2pnC7iqSetsBrd6Y+Um/kquOHukfHiNNvR23O6Qfk4N+44RpInXWdDSHU4BIPhVsRmbkf+eMbHOqEE02nt5addulbJ8UoPQhyFYUsXXDSgRtXKF8p4K2DLleIgEJId5IsGWmGApFaf27bdmq9F+Z2PME3n+L8Oru2nFy/wg1+HDNc8m2Kw6LnSO8Yfkr+qezByZEo4f0mI9O8aBqzitD+a1CvBJfKttQNXQCUoJ4ckb+BAJGr2CusTXkgdCEEK9s7G6OmQqGhUrY0LRXTYI9/i8LabRza9xehRPAq/ZdqkKQXMo2UL6QyZZfITKdCuZePqPl/NRchc6AIH7VYII1yZWl5ksTttgc7BJpa3FdBYOuO7nIfMgIEgB76b5QsmEkVp4AZ7KlBv7lvelkLa8pp2UHt+HQHKx00Lm9tZ64evQtxv5Tkm0J9glkmBfEmQdvDA2E7FoX+nW/2ZgIGPgYLkrcgyej+qMBarzdDRZvQtjgqZtMhdKH7BU= ceph-27e8cf32-043a-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T02:04:09.511 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHhVyYxV3l2pnC7iqSetsBrd6Y+Um/kquOHukfHiNNvR23O6Qfk4N+44RpInXWdDSHU4BIPhVsRmbkf+eMbHOqEE02nt5addulbJ8UoPQhyFYUsXXDSgRtXKF8p4K2DLleIgEJId5IsGWmGApFaf27bdmq9F+Z2PME3n+L8Oru2nFy/wg1+HDNc8m2Kw6LnSO8Yfkr+qezByZEo4f0mI9O8aBqzitD+a1CvBJfKttQNXQCUoJ4ckb+BAJGr2CusTXkgdCEEK9s7G6OmQqGhUrY0LRXTYI9/i8LabRza9xehRPAq/ZdqkKQXMo2UL6QyZZfITKdCuZePqPl/NRchc6AIH7VYII1yZWl5ksTttgc7BJpa3FdBYOuO7nIfMgIEgB76b5QsmEkVp4AZ7KlBv7lvelkLa8pp2UHt+HQHKx00Lm9tZ64evQtxv5Tkm0J9glkmBfEmQdvDA2E7FoX+nW/2ZgIGPgYLkrcgyej+qMBarzdDRZvQtjgqZtMhdKH7BU= ceph-27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:04:09.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-27T02:04:09.544 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:09 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/392547484' entity='client.admin' 2024-04-27T02:04:12.918 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:12 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3710682921' entity='client.admin' 2024-04-27T02:04:12.919 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:12 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:12.919 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:12 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:04:12.919 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:12 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:12.942 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-27T02:04:12.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-27T02:04:13.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:13 smithi008 ceph-mon[21873]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-27T02:04:16.250 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2024-04-27T02:04:16.250 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T02:04:16.250 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2024-04-27T02:04:16.269 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T02:04:16.269 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:04:16.326 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2024-04-27T02:04:16.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch host add smithi077 2024-04-27T02:04:16.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:16 smithi008 ceph-mon[21873]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:16.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:16 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:19.668 INFO:teuthology.orchestra.run.smithi008.stdout:Added host 'smithi077' with addr '172.21.15.77' 2024-04-27T02:04:20.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:20 smithi008 ceph-mon[21873]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:20.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:20 smithi008 ceph-mon[21873]: Deploying cephadm binary to smithi077 2024-04-27T02:04:20.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:20 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:20.764 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:20 smithi008 ceph-mon[21873]: Added host smithi077 2024-04-27T02:04:21.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-27T02:04:21.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:21 smithi008 ceph-mon[21873]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:23.853 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:04:23.853 INFO:teuthology.orchestra.run.smithi008.stdout:[{"addr": "172.21.15.8", "hostname": "smithi008", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2024-04-27T02:04:24.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:23 smithi008 ceph-mon[21873]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:24.887 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-27T02:04:24.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-27T02:04:25.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:04:25.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:25.717 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:25.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:04:25.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T02:04:25.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:25 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:26.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:26 smithi008 ceph-mon[21873]: Deploying daemon crash.smithi008 on smithi008 2024-04-27T02:04:27.683 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-04-27T02:04:28.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:27 smithi008 ceph-mon[21873]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:28.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:27 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1055918043' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T02:04:28.352 INFO:tasks.cephadm:Adding mon.smithi008 on smithi008 2024-04-27T02:04:28.352 INFO:tasks.cephadm:Adding mon.smithi077 on smithi077 2024-04-27T02:04:28.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi008:172.21.15.8=smithi008;smithi077:172.21.15.77=smithi077' 2024-04-27T02:04:29.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:28 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1055918043' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T02:04:29.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:28 smithi008 ceph-mon[21873]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T02:04:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:29 smithi008 ceph-mon[21873]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:29.757 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:29 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:30.209 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mon update... 2024-04-27T02:04:30.792 DEBUG:teuthology.orchestra.run.smithi077:mon.smithi077> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi077.service 2024-04-27T02:04:30.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:30.794 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:04:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:04:31.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:31.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: Deploying daemon grafana.smithi008 on smithi008 2024-04-27T02:04:31.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:31.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:30 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:32.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:31 smithi008 ceph-mon[21873]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:32.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:31 smithi008 ceph-mon[21873]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi008:172.21.15.8=smithi008;smithi077:172.21.15.77=smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:04:32.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:31 smithi008 ceph-mon[21873]: Saving service mon spec with placement smithi008:172.21.15.8=smithi008;smithi077:172.21.15.77=smithi077;count:2 2024-04-27T02:04:32.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:32.650 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:32.652 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:33.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:32 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/3174228723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:34.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:34.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:34.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:33 smithi008 ceph-mon[21873]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:35.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:35 smithi008 ceph-mon[21873]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:35.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:35.967 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:35.969 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:37.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:36 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/164381369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:37.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:37.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:38.061 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:37 smithi008 ceph-mon[21873]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:39.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:39.377 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:39.378 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:40.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:39 smithi008 ceph-mon[21873]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:40.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:39 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/3930844650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:40.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:40.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:42.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:41 smithi008 ceph-mon[21873]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:42.811 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:42.811 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:42.813 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:44.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:43 smithi008 ceph-mon[21873]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:44.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:43 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:44.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:43 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/590650115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:44.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:44.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:44.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:44 smithi008 ceph-mon[21873]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-27T02:04:46.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:45 smithi008 ceph-mon[21873]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:46.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:45 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:46.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:46.166 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:47.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:46 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/3489825104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:47.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:47.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:48.000 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:47 smithi008 ceph-mon[21873]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:49 smithi008 ceph-mon[21873]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:04:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:49 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:49 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T02:04:49.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:49.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:49.595 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:50.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:50 smithi008 ceph-mon[21873]: from='mgr.14162 172.21.15.8:0/2249657569' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T02:04:50.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:50 smithi008 ceph-mon[21873]: mgrmap e14: smithi008.exkhsc(active, since 49s) 2024-04-27T02:04:50.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:50 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2173016508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:51.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:53.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:53.024 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:53.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:53 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/3639171654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:54.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:54.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:56.485 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:04:56.486 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:04:56.487 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:04:56.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: Active manager daemon smithi008.exkhsc restarted 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: mgrmap e15: smithi008.exkhsc(active, starting, since 0.0524766s) 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:04:56.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:56.662 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:57.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/733949849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: mgrmap e16: smithi008.exkhsc(active, since 1.05518s) 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:56] ENGINE Bus STARTING 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:56] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: [27/Apr/2024:02:04:56] ENGINE Bus STARTED 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:57.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:57 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:04:58.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:04:58.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:04:59.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:04:59 smithi008 ceph-mon[21873]: mgrmap e17: smithi008.exkhsc(active, since 2s) 2024-04-27T02:05:01.284 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:05:01.284 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:05:01.286 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:05:02.008 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:02.009 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:02.009 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:05:02.009 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:01 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/1245063030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:05:02.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:05:02.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:05:04.694 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:05:04.694 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:05:04.696 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:05:04.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T02:05:04.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:05.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:05 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:05:05.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:05 smithi008 ceph-mon[21873]: Deploying daemon crash.smithi077 on smithi077 2024-04-27T02:05:05.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:05 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/737878442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:05:06.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:05:06.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:05:08.348 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:05:08.348 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:05:08.349 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:05:08.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:08 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/748033882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:05:09.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:09.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:05:09.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T02:05:09.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:05:09.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:09.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:09 smithi008 ceph-mon[21873]: Deploying daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:05:10.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:05:10.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:05:11.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:11 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:12.121 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:05:12.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:02:59.719455Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T02:05:12.123 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-04-27T02:05:12.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:12 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/369488014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:05:13.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T02:05:13.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T02:05:14.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:05:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:14 smithi008 ceph-mon[21873]: Deploying daemon mon.smithi077 on smithi077 2024-04-27T02:05:16.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:16 smithi008 ceph-mon[21873]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:16.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:22.107 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T02:05:22.107 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","modified":"2024-04-27T02:05:17.114745Z","created":"2024-04-27T02:02:59.719455Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi008","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-27T02:05:22.109 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 2 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2997816867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: mon.smithi008 calling monitor election 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/1415291188' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: mon.smithi077 calling monitor election 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: mon.smithi008 is new leader, mons smithi008,smithi077 in quorum (ranks 0,1) 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: monmap e2: 2 mons at {smithi008=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: fsmap 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: mgrmap e17: smithi008.exkhsc(active, since 26s) 2024-04-27T02:05:22.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:05:22.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/1415291188' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:05:22.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: overall HEALTH_OK 2024-04-27T02:05:22.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/1415291188' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:05:22.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/1415291188' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:05:22.413 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:22.715 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-27T02:05:22.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-27T02:05:23.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:23 smithi008 ceph-mon[21873]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:23.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:23 smithi008 ceph-mon[21873]: Deploying daemon node-exporter.smithi077 on smithi077 2024-04-27T02:05:23.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:23 smithi008 ceph-mon[21873]: mgrmap e18: smithi008.exkhsc(active, since 26s), standbys: smithi077.snksrp 2024-04-27T02:05:23.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:23 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:05:23.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:23 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:05:24.558 INFO:teuthology.orchestra.run.smithi008.stdout:# minimal ceph.conf for 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:05:24.559 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-04-27T02:05:24.559 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:05:24.559 INFO:teuthology.orchestra.run.smithi008.stdout: mon_host = [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2024-04-27T02:05:25.039 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-27T02:05:25.040 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:05:25.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T02:05:25.067 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:05:25.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:05:25.133 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T02:05:25.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T02:05:25.162 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T02:05:25.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:05:25.227 INFO:tasks.cephadm:Deploying OSDs... 2024-04-27T02:05:25.227 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:05:25.227 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T02:05:25.243 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T02:05:25.243 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-04-27T02:05:25.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:25 smithi008 ceph-mon[21873]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:25.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:25 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/4006837159' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-27 02:04:43.485722646 +0000 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-27 02:01:50.462977449 +0000 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-27 02:01:50.462977449 +0000 2024-04-27T02:05:25.298 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-27 02:01:50.462977449 +0000 2024-04-27T02:05:25.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T02:05:25.362 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-27T02:05:25.362 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-27T02:05:25.362 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000116962 s, 4.4 MB/s 2024-04-27T02:05:25.363 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T02:05:25.420 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-04-27T02:05:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T02:05:25.477 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:25.477 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-27 02:04:43.485722646 +0000 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-27 02:01:50.803973770 +0000 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-27 02:01:50.803973770 +0000 2024-04-27T02:05:25.478 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-27 02:01:50.803973770 +0000 2024-04-27T02:05:25.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T02:05:25.541 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-27T02:05:25.541 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-27T02:05:25.541 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000106949 s, 4.8 MB/s 2024-04-27T02:05:25.543 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T02:05:25.598 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:25.654 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-27 02:04:43.485722646 +0000 2024-04-27T02:05:25.655 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-27 02:01:51.145970081 +0000 2024-04-27T02:05:25.655 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-27 02:01:51.145970081 +0000 2024-04-27T02:05:25.655 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-27 02:01:51.145970081 +0000 2024-04-27T02:05:25.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T02:05:25.717 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-27T02:05:25.717 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-27T02:05:25.717 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.0001071 s, 4.8 MB/s 2024-04-27T02:05:25.718 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T02:05:25.775 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-04-27T02:05:25.829 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-04-27 02:04:43.486722635 +0000 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-04-27 02:01:51.483966435 +0000 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-04-27 02:01:51.483966435 +0000 2024-04-27T02:05:25.830 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-04-27 02:01:51.483966435 +0000 2024-04-27T02:05:25.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T02:05:25.892 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-04-27T02:05:25.893 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-04-27T02:05:25.893 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 9.3226e-05 s, 5.5 MB/s 2024-04-27T02:05:25.894 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T02:05:25.952 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T02:05:25.952 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T02:05:25.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T02:05:25.967 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-04-27T02:05:26.022 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T02:05:26.022 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:26.022 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-27T02:05:26.022 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:26.022 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:26.023 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 02:05:23.022744699 +0000 2024-04-27T02:05:26.023 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 02:01:43.336775640 +0000 2024-04-27T02:05:26.023 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 02:01:43.336775640 +0000 2024-04-27T02:05:26.023 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-27 02:01:43.336775640 +0000 2024-04-27T02:05:26.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T02:05:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T02:05:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T02:05:26.085 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00022111 s, 2.3 MB/s 2024-04-27T02:05:26.086 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T02:05:26.142 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-04-27T02:05:26.197 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 02:05:23.023744678 +0000 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 02:01:43.675767798 +0000 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 02:01:43.675767798 +0000 2024-04-27T02:05:26.198 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-27 02:01:43.675767798 +0000 2024-04-27T02:05:26.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T02:05:26.259 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T02:05:26.259 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T02:05:26.259 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00017517 s, 2.9 MB/s 2024-04-27T02:05:26.260 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T02:05:26.316 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-04-27T02:05:26.370 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T02:05:26.370 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:26.370 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-27T02:05:26.370 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:26.371 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:26.371 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 02:05:23.023744678 +0000 2024-04-27T02:05:26.371 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 02:01:44.009760072 +0000 2024-04-27T02:05:26.371 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 02:01:44.009760072 +0000 2024-04-27T02:05:26.371 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-27 02:01:44.009760072 +0000 2024-04-27T02:05:26.371 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T02:05:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T02:05:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T02:05:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000246944 s, 2.1 MB/s 2024-04-27T02:05:26.434 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T02:05:26.489 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-04-27T02:05:26.543 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T02:05:26.543 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 02:05:23.024744657 +0000 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 02:01:44.345752299 +0000 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 02:01:44.345752299 +0000 2024-04-27T02:05:26.544 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-04-27 02:01:44.345752299 +0000 2024-04-27T02:05:26.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T02:05:26.606 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T02:05:26.607 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T02:05:26.607 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000170739 s, 3.0 MB/s 2024-04-27T02:05:26.608 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T02:05:26.663 INFO:tasks.cephadm:Deploying osd.0 on smithi008 with /dev/vg_nvme/lv_4... 2024-04-27T02:05:26.663 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T02:05:26.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:26 smithi008 ceph-mon[21873]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:26.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:26 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:26.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:26 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:26.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:26 smithi008 ceph-mon[21873]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-27T02:05:29.366 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T02:05:29.366 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T02:05:29.367 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-27T02:05:29.367 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-27T02:05:29.367 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00869764 s, 1.2 GB/s 2024-04-27T02:05:29.367 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-27T02:05:29.367 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:29 smithi008 ceph-mon[21873]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:29.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:29 smithi077 ceph-mon[24318]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:30.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_4 2024-04-27T02:05:31.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:31 smithi008 ceph-mon[21873]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:31.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:31 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:31.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:31 smithi077 ceph-mon[24318]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:31.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:31 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:33 smithi008 ceph-mon[21873]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:33.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:33 smithi077 ceph-mon[24318]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:35.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:35 smithi008 ceph-mon[21873]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:35.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:05:35.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:05:35.369 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:35 smithi077 ceph-mon[24318]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:05:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:05:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:36.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:36 smithi008 ceph-mon[21873]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:36.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:36 smithi077 ceph-mon[24318]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:05:37.383 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:37 smithi008 ceph-mon[21873]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:37.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:37 smithi077 ceph-mon[24318]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:39.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:39 smithi077 ceph-mon[24318]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:39.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:39 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:39.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:39 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:39.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:39 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:39.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:39 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:05:39.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:39 smithi008 ceph-mon[21873]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:39.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:39 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:39.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:39 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:05:39.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:39 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:39.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:39 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:05:40.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:40 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/4020385782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "426fa516-32ce-4ac8-90f6-fb5cf5ed20ce"}]: dispatch 2024-04-27T02:05:40.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:40 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/4020385782' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "426fa516-32ce-4ac8-90f6-fb5cf5ed20ce"}]': finished 2024-04-27T02:05:40.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:40 smithi077 ceph-mon[24318]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T02:05:40.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:40 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:05:40.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:40 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/4020385782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "426fa516-32ce-4ac8-90f6-fb5cf5ed20ce"}]: dispatch 2024-04-27T02:05:40.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:40 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/4020385782' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "426fa516-32ce-4ac8-90f6-fb5cf5ed20ce"}]': finished 2024-04-27T02:05:40.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:40 smithi008 ceph-mon[21873]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T02:05:40.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:40 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:05:41.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:41 smithi077 ceph-mon[24318]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:41.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:41.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:41.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:41 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2293228760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:05:41.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:41.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:41 smithi008 ceph-mon[21873]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:41.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:41.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:41.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:41 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2293228760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:05:41.408 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:42.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:42 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:05:42.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:42 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:05:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:42 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:05:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:42 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:05:43.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:43 smithi008 ceph-mon[21873]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:43.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:43 smithi077 ceph-mon[24318]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:44.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:44.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:44 smithi077 ceph-mon[24318]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: Reconfiguring mon.smithi008 (unknown last config time)... 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:44 smithi008 ceph-mon[21873]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-27T02:05:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:45 smithi077 ceph-mon[24318]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:05:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:05:45.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:45 smithi008 ceph-mon[21873]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:45.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:05:45.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:05:45.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:46.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:46 smithi008 ceph-mon[21873]: Reconfiguring mgr.smithi008.exkhsc (unknown last config time)... 2024-04-27T02:05:46.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:46 smithi008 ceph-mon[21873]: Reconfiguring daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:05:46.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:46 smithi077 ceph-mon[24318]: Reconfiguring mgr.smithi008.exkhsc (unknown last config time)... 2024-04-27T02:05:46.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:46 smithi077 ceph-mon[24318]: Reconfiguring daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:05:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:47 smithi008 ceph-mon[21873]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:47 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:47.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:47 smithi077 ceph-mon[24318]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:47.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:47 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:48.456 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:48 smithi008 ceph-mon[21873]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-27T02:05:48.456 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:48 smithi008 ceph-mon[21873]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-27T02:05:48.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:48 smithi077 ceph-mon[24318]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-27T02:05:48.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:48 smithi077 ceph-mon[24318]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-27T02:05:49.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:49 smithi077 ceph-mon[24318]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:49.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:05:49.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:49.609 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:49 smithi008 ceph-mon[21873]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:49.609 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:05:49.609 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:50.224 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:50 smithi008 ceph-mon[21873]: Deploying daemon osd.0 on smithi008 2024-04-27T02:05:50.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:50 smithi077 ceph-mon[24318]: Deploying daemon osd.0 on smithi008 2024-04-27T02:05:51.207 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:51 smithi008 ceph-mon[21873]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:51.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:51 smithi077 ceph-mon[24318]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:53.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:53 smithi008 ceph-mon[21873]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:53.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:53 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:53.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:53 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:05:53.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:53 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:53.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:53 smithi077 ceph-mon[24318]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:53.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:53 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:53.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:53 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:05:53.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:53 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:05:54.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:54 smithi077 ceph-mon[24318]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-27T02:05:54.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:54 smithi077 ceph-mon[24318]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-27T02:05:54.529 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:54 smithi008 ceph-mon[21873]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-27T02:05:54.529 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:54 smithi008 ceph-mon[21873]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-27T02:05:55.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:55 smithi008 ceph-mon[21873]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:55.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:55 smithi077 ceph-mon[24318]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:56.381 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:05:56.381 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:05:56.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:05:56.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:05:57.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:57 smithi008 ceph-mon[21873]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:57.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:57 smithi077 ceph-mon[24318]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:59.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:59 smithi008 ceph-mon[21873]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:59.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:05:59 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:05:59.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:59 smithi077 ceph-mon[24318]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:05:59.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:05:59 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:01.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:01 smithi008 ceph-mon[21873]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:01 smithi008 ceph-mon[21873]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-27T02:06:01.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:01 smithi008 ceph-mon[21873]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-27T02:06:01.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:01 smithi077 ceph-mon[24318]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:01.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:01.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:01 smithi077 ceph-mon[24318]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-27T02:06:01.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:01 smithi077 ceph-mon[24318]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-27T02:06:02.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:02 smithi008 ceph-mon[21873]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:06:02.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:02 smithi077 ceph-mon[24318]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:06:03.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:03 smithi008 ceph-mon[21873]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:03.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:03 smithi008 ceph-mon[21873]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:06:03.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:03 smithi008 ceph-mon[21873]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T02:06:03.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:03.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:03 smithi008 ceph-mon[21873]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:06:03.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:03 smithi077 ceph-mon[24318]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:03.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:03 smithi077 ceph-mon[24318]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:06:03.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:03 smithi077 ceph-mon[24318]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T02:06:03.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:03.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:03 smithi077 ceph-mon[24318]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:06:04.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:04 smithi008 ceph-mon[21873]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:06:04.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:04 smithi008 ceph-mon[21873]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T02:06:04.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:04.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:04.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:04 smithi077 ceph-mon[24318]: from='osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:06:04.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:04 smithi077 ceph-mon[24318]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T02:06:04.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:04.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476] boot 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:06:05.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:05.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: osd.0 [v2:172.21.15.8:6802/425614476,v1:172.21.15.8:6803/425614476] boot 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:06:05.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:05.980 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 0 on host 'smithi008' 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: Reconfiguring crash.smithi077 (monmap changed)... 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: Reconfiguring daemon crash.smithi077 on smithi077 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:06:06.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:06.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: Reconfiguring crash.smithi077 (monmap changed)... 2024-04-27T02:06:06.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: Reconfiguring daemon crash.smithi077 on smithi077 2024-04-27T02:06:06.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T02:06:06.302 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:06.302 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:06.302 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:06:06.302 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:06:06.302 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:06.594 DEBUG:teuthology.orchestra.run.smithi008:osd.0> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.0.service 2024-04-27T02:06:06.597 INFO:tasks.cephadm:Deploying osd.1 on smithi008 with /dev/vg_nvme/lv_3... 2024-04-27T02:06:06.597 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T02:06:07.272 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:07 smithi008 ceph-mon[21873]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:07.272 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:07 smithi008 ceph-mon[21873]: Reconfiguring mgr.smithi077.snksrp (monmap changed)... 2024-04-27T02:06:07.272 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:07 smithi008 ceph-mon[21873]: Reconfiguring daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:06:07.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:07 smithi077 ceph-mon[24318]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:07.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:07 smithi077 ceph-mon[24318]: Reconfiguring mgr.smithi077.snksrp (monmap changed)... 2024-04-27T02:06:07.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:07 smithi077 ceph-mon[24318]: Reconfiguring daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: Reconfiguring mon.smithi077 (monmap changed)... 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: Reconfiguring daemon mon.smithi077 on smithi077 2024-04-27T02:06:08.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:08 smithi008 ceph-mon[21873]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: Reconfiguring mon.smithi077 (monmap changed)... 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: Reconfiguring daemon mon.smithi077 on smithi077 2024-04-27T02:06:08.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:08 smithi077 ceph-mon[24318]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00889064 s, 1.2 GB/s 2024-04-27T02:06:08.728 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-27T02:06:09.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_3 2024-04-27T02:06:09.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:06:09.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-27T02:06:09.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.607 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:09.607 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:09.607 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.8:9093"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-27T02:06:09.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.8:3000"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.8:9095"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:09.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:10.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:10 smithi008 ceph-mon[21873]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:10.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:10 smithi077 ceph-mon[24318]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:12.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:12 smithi008 ceph-mon[21873]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:12.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:12 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:06:12.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:12 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:06:12.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:12 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:12.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:12 smithi077 ceph-mon[24318]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:12.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:12 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:06:12.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:12 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:06:12.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:12 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:13.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:13 smithi008 ceph-mon[21873]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:13.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:13 smithi077 ceph-mon[24318]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:15.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:15 smithi008 ceph-mon[21873]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:15.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:15 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:15.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:15 smithi077 ceph-mon[24318]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:15.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:15 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:17.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:17.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1811623363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f83e4d6-fbf7-449b-aaef-f60f5a68641e"}]: dispatch 2024-04-27T02:06:17.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1811623363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f83e4d6-fbf7-449b-aaef-f60f5a68641e"}]': finished 2024-04-27T02:06:17.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T02:06:17.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:17.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:17 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1343989941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1811623363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f83e4d6-fbf7-449b-aaef-f60f5a68641e"}]: dispatch 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1811623363' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f83e4d6-fbf7-449b-aaef-f60f5a68641e"}]': finished 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:17.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:17 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1343989941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:06:19.311 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:19 smithi008 ceph-mon[21873]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:19.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:19 smithi077 ceph-mon[24318]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:21.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:21 smithi008 ceph-mon[21873]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:21 smithi077 ceph-mon[24318]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi008 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi008 to 12869M 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:22.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi008 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi008 to 12869M 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:23.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:23.062 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:23.062 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:24.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:23 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:06:24.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:23 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:24.107 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:23 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:24.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:23 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:06:24.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:23 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:24.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:23 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:25.091 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:24 smithi008 ceph-mon[21873]: Deploying daemon osd.1 on smithi008 2024-04-27T02:06:25.091 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:24 smithi008 ceph-mon[21873]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:25.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:24 smithi077 ceph-mon[24318]: Deploying daemon osd.1 on smithi008 2024-04-27T02:06:25.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:24 smithi077 ceph-mon[24318]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:26.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:26 smithi008 ceph-mon[21873]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:26 smithi077 ceph-mon[24318]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:29.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:28 smithi008 ceph-mon[21873]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:28 smithi077 ceph-mon[24318]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:30.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:30.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:30.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:30 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:30.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:30 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:31.375 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:31 smithi008 ceph-mon[21873]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:31.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:31 smithi077 ceph-mon[24318]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:33.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:32 smithi008 ceph-mon[21873]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:33.137 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:33.227 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 1 on host 'smithi008' 2024-04-27T02:06:33.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:32 smithi077 ceph-mon[24318]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:33.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:33 smithi008 ceph-mon[21873]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:06:33.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:33.825 DEBUG:teuthology.orchestra.run.smithi008:osd.1> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.1.service 2024-04-27T02:06:33.828 INFO:tasks.cephadm:Deploying osd.2 on smithi008 with /dev/vg_nvme/lv_2... 2024-04-27T02:06:33.828 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T02:06:34.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:33 smithi077 ceph-mon[24318]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:06:34.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:35.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-mon[21873]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:35.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-mon[21873]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:06:35.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-mon[21873]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T02:06:35.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:35.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-mon[21873]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:06:35.161 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[36412]: 2024-04-27T02:06:34.820+0000 7fdfd2ffc700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T02:06:35.161 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:06:34 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[36412]: 2024-04-27T02:06:34.824+0000 7fdfcd993700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:06:35.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:34 smithi077 ceph-mon[24318]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:35.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:34 smithi077 ceph-mon[24318]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:06:35.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:34 smithi077 ceph-mon[24318]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T02:06:35.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:34 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:35.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:34 smithi077 ceph-mon[24318]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:06:35.856 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:35 smithi008 ceph-mon[21873]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:06:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:35 smithi008 ceph-mon[21873]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T02:06:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:35.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:36.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:35 smithi077 ceph-mon[24318]: from='osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:06:36.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:35 smithi077 ceph-mon[24318]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T02:06:36.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:36.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:36.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:36.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:06:36.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:06:36.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:36.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045] boot 2024-04-27T02:06:36.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T02:06:36.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:36 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00988472 s, 1.1 GB/s 2024-04-27T02:06:36.983 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: osd.1 [v2:172.21.15.8:6810/2674833045,v1:172.21.15.8:6811/2674833045] boot 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T02:06:37.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:36 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:06:37.621 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_2 2024-04-27T02:06:37.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:37 smithi008 ceph-mon[21873]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T02:06:38.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:37 smithi077 ceph-mon[24318]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T02:06:39.137 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:38 smithi077 ceph-mon[24318]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:39.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:38 smithi008 ceph-mon[21873]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:06:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:06:41.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:41.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:41.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:41.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:06:41.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:06:41.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:06:41.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:43.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:43.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:44.279 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:44 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi008 to 6434M 2024-04-27T02:06:44.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:44 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi008 to 6434M 2024-04-27T02:06:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:45 smithi008 ceph-mon[21873]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:45 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3242296688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8081e46-868d-4256-8838-462607a616b6"}]: dispatch 2024-04-27T02:06:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:45 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3242296688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8081e46-868d-4256-8838-462607a616b6"}]': finished 2024-04-27T02:06:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:45 smithi008 ceph-mon[21873]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T02:06:45.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:06:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:45 smithi077 ceph-mon[24318]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:45 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3242296688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8081e46-868d-4256-8838-462607a616b6"}]: dispatch 2024-04-27T02:06:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:45 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3242296688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8081e46-868d-4256-8838-462607a616b6"}]': finished 2024-04-27T02:06:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:45 smithi077 ceph-mon[24318]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T02:06:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:06:46.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:46 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2831121007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:06:46.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:46 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2831121007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:06:47.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:47 smithi008 ceph-mon[21873]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:47.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:47 smithi077 ceph-mon[24318]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:48.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:48 smithi008 ceph-mon[21873]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:48.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:48 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:48.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:48 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:48.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:48 smithi077 ceph-mon[24318]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:48.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:48 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:48.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:48 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:50.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:50 smithi008 ceph-mon[21873]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:50 smithi077 ceph-mon[24318]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:51.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:06:51.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:51.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:51 smithi077 ceph-mon[24318]: Deploying daemon osd.2 on smithi008 2024-04-27T02:06:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:06:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:51 smithi008 ceph-mon[21873]: Deploying daemon osd.2 on smithi008 2024-04-27T02:06:52.857 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:52 smithi008 ceph-mon[21873]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:52.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:52 smithi077 ceph-mon[24318]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:54.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:54 smithi077 ceph-mon[24318]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:55.126 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:54 smithi008 ceph-mon[21873]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:55.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:06:55.870 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:06:55.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:06:55.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:06:57.203 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:56 smithi008 ceph-mon[21873]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:57.203 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:57.203 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:57.203 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:57.203 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:56 smithi077 ceph-mon[24318]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:06:57.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:06:57.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:06:58.635 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 2 on host 'smithi008' 2024-04-27T02:06:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:58 smithi077 ceph-mon[24318]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:58 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:59.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:58 smithi008 ceph-mon[21873]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:06:59.250 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:58 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:06:59.543 DEBUG:teuthology.orchestra.run.smithi008:osd.2> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.2.service 2024-04-27T02:06:59.546 INFO:tasks.cephadm:Deploying osd.3 on smithi008 with /dev/vg_nvme/lv_1... 2024-04-27T02:06:59.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T02:07:00.103 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:06:59 smithi008 ceph-mon[21873]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:07:00.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:06:59 smithi077 ceph-mon[24318]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:07:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-mon[21873]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:07:01.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-mon[21873]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:07:01.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-mon[21873]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T02:07:01.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:01.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-mon[21873]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:07:01.161 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[40123]: 2024-04-27T02:07:00.952+0000 7f2bc759b700 -1 osd.2 0 waiting for initial osdmap 2024-04-27T02:07:01.161 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:00 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[40123]: 2024-04-27T02:07:00.956+0000 7f2bc1f32700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:07:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:00 smithi077 ceph-mon[24318]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:07:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:00 smithi077 ceph-mon[24318]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:07:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:00 smithi077 ceph-mon[24318]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T02:07:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:00 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:00 smithi077 ceph-mon[24318]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:07:02.142 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:01 smithi008 ceph-mon[21873]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:07:02.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:01 smithi008 ceph-mon[21873]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T02:07:02.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:02.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:02.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:01 smithi077 ceph-mon[24318]: from='osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:07:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:01 smithi077 ceph-mon[24318]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T02:07:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:03.069 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T02:07:03.069 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T02:07:03.069 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-04-27T02:07:03.069 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-04-27T02:07:03.069 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109711 s, 956 MB/s 2024-04-27T02:07:03.070 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746] boot 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T02:07:03.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:02 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: osd.2 [v2:172.21.15.8:6818/896710746,v1:172.21.15.8:6819/896710746] boot 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T02:07:03.377 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:02 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:07:04.019 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi008:vg_nvme/lv_1 2024-04-27T02:07:04.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:04 smithi008 ceph-mon[21873]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T02:07:04.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:04 smithi077 ceph-mon[24318]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T02:07:05.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:05 smithi008 ceph-mon[21873]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:05.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-27T02:07:05.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-27T02:07:05.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:05 smithi008 ceph-mon[21873]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T02:07:05.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:05 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:07:05.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:05 smithi077 ceph-mon[24318]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:05.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-27T02:07:05.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-27T02:07:05.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:05 smithi077 ceph-mon[24318]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T02:07:05.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:05 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T02:07:06.346 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:06 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T02:07:06.346 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:06 smithi008 ceph-mon[21873]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T02:07:06.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:06 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T02:07:06.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:06 smithi077 ceph-mon[24318]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T02:07:06.910 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:06 smithi008 sudo[41679]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:07:06.910 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:06 smithi008 sudo[41679]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:06.910 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:06 smithi008 sudo[41679]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:07.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:07.303 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: osdmap e23: 3 total, 3 up, 3 in 2024-04-27T02:07:07.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:07.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:07.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:07.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:07 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:07.304 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41739]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:07:07.304 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41739]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:07.304 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41739]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: osdmap e23: 3 total, 3 up, 3 in 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:07.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:07 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:07.660 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41760]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:07:07.660 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41760]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:07.660 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:07 smithi008 sudo[41760]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:08.314 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:08 smithi008 ceph-mon[21873]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:08.314 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:08 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:08.314 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41786]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:07:08.314 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41786]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:08.315 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41786]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:08.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:08 smithi077 ceph-mon[24318]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:08.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:08 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:08.564 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41808]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:07:08.565 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41808]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:08.565 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41808]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:09.160 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41859]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:07:09.160 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41859]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:09.160 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:07:08 smithi008 sudo[41859]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:09.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41865]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:07:09.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41865]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:09.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41865]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41879]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41879]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 sudo[41879]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 ceph-mon[21873]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 ceph-mon[21873]: mgrmap e19: smithi008.exkhsc(active, since 2m), standbys: smithi077.snksrp 2024-04-27T02:07:10.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:09 smithi008 ceph-mon[21873]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:07:10.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:09 smithi077 ceph-mon[24318]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:10.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:09 smithi077 ceph-mon[24318]: mgrmap e19: smithi008.exkhsc(active, since 2m), standbys: smithi077.snksrp 2024-04-27T02:07:10.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:09 smithi077 ceph-mon[24318]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:07:10.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26523]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T02:07:10.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26523]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:10.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26523]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:10.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26526]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T02:07:10.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26526]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T02:07:10.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 sudo[26526]: pam_unix(sudo:session): session closed for user root 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi008 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi008 to 4289M 2024-04-27T02:07:10.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:10.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:10 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi008 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:07:10.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi008 to 4289M 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:10.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:10 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:12.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:11 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1550936181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b"}]: dispatch 2024-04-27T02:07:12.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:11 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1550936181' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b"}]': finished 2024-04-27T02:07:12.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:11 smithi008 ceph-mon[21873]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T02:07:12.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:11 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:12.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:11 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/509352173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:07:12.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:11 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1550936181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b"}]: dispatch 2024-04-27T02:07:12.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:11 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1550936181' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b"}]': finished 2024-04-27T02:07:12.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:11 smithi077 ceph-mon[24318]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T02:07:12.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:11 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:12.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:11 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/509352173' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:07:13.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:12 smithi077 ceph-mon[24318]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:12 smithi008 ceph-mon[21873]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:15.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:14 smithi008 ceph-mon[21873]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-04-27T02:07:15.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:14 smithi077 ceph-mon[24318]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-04-27T02:07:17.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:16 smithi077 ceph-mon[24318]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-04-27T02:07:17.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:16 smithi008 ceph-mon[21873]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-04-27T02:07:17.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:17 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:07:17.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:17 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:18.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:17 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:07:18.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:17 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:19.068 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:18 smithi008 ceph-mon[21873]: Deploying daemon osd.3 on smithi008 2024-04-27T02:07:19.068 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:18 smithi008 ceph-mon[21873]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:07:19.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:18 smithi077 ceph-mon[24318]: Deploying daemon osd.3 on smithi008 2024-04-27T02:07:19.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:18 smithi077 ceph-mon[24318]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:07:21.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:20 smithi077 ceph-mon[24318]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:07:21.260 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:20 smithi008 ceph-mon[21873]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-27T02:07:22.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:22 smithi077 ceph-mon[24318]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 123 KiB/s wr, 8 op/s 2024-04-27T02:07:22.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:22.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:22.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:22.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:22 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:23.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:22 smithi008 ceph-mon[21873]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 123 KiB/s wr, 8 op/s 2024-04-27T02:07:23.045 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:23.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:23.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:23.046 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:22 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:25.123 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:24 smithi008 ceph-mon[21873]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-27T02:07:25.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:24 smithi077 ceph-mon[24318]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-27T02:07:26.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:25 smithi008 ceph-mon[21873]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:07:26.082 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 3 on host 'smithi008' 2024-04-27T02:07:26.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:25 smithi077 ceph-mon[24318]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:07:27.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:07:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T02:07:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:07:27.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:26 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:07:27.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:26 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:27.299 DEBUG:teuthology.orchestra.run.smithi008:osd.3> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.3.service 2024-04-27T02:07:27.302 INFO:tasks.cephadm:Deploying osd.4 on smithi077 with /dev/vg_nvme/lv_4... 2024-04-27T02:07:27.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T02:07:28.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:27 smithi008 ceph-mon[21873]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:07:28.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:27 smithi008 ceph-mon[21873]: osdmap e26: 4 total, 3 up, 4 in 2024-04-27T02:07:28.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:28.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:27 smithi077 ceph-mon[24318]: from='osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-04-27T02:07:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:27 smithi077 ceph-mon[24318]: osdmap e26: 4 total, 3 up, 4 in 2024-04-27T02:07:28.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:28.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236] boot 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:29.145 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:28 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: osd.3 [v2:172.21.15.8:6826/3468489236,v1:172.21.15.8:6827/3468489236] boot 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:07:29.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:28 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:29.363 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T02:07:29.363 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T02:07:29.363 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-27T02:07:29.363 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-27T02:07:29.364 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148247 s, 707 MB/s 2024-04-27T02:07:29.364 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-27T02:07:29.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-04-27T02:07:30.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:29 smithi008 ceph-mon[21873]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T02:07:30.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:29 smithi077 ceph-mon[24318]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T02:07:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:30 smithi008 ceph-mon[21873]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:31.222 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:30 smithi077 ceph-mon[24318]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:32.227 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:31 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:32.227 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:31 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:32.227 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:31 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:32.270 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:31 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:32.271 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:31 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:32.271 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:31 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:33.116 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:33.117 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='client.24147 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:33.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:34.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:34 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi008 2024-04-27T02:07:34.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:34 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi008 to 3217M 2024-04-27T02:07:34.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:34 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi008 2024-04-27T02:07:34.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:34 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi008 to 3217M 2024-04-27T02:07:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2416396248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]: dispatch 2024-04-27T02:07:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]: dispatch 2024-04-27T02:07:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]': finished 2024-04-27T02:07:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T02:07:35.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:35.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:35 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2996438015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:07:35.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/2416396248' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]: dispatch 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]: dispatch 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "248d0710-cd99-4824-86c1-f1b93e727cb5"}]': finished 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:35.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:35 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/2996438015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:07:37.174 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:37 smithi077 ceph-mon[24318]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:37.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:37 smithi008 ceph-mon[21873]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:39.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:39 smithi077 ceph-mon[24318]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:39.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:39 smithi008 ceph-mon[21873]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:41 smithi008 ceph-mon[21873]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:07:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:41 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:41.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:41 smithi008 ceph-mon[21873]: Deploying daemon osd.4 on smithi077 2024-04-27T02:07:41.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:41 smithi077 ceph-mon[24318]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:41.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:07:41.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:41 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:41.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:41 smithi077 ceph-mon[24318]: Deploying daemon osd.4 on smithi077 2024-04-27T02:07:43.247 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:43 smithi077 ceph-mon[24318]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:43.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:43 smithi008 ceph-mon[21873]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:45.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:45 smithi077 ceph-mon[24318]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:45.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:45.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:45.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:45.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:45 smithi008 ceph-mon[21873]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:45.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:47.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:47 smithi077 ceph-mon[24318]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:47 smithi008 ceph-mon[21873]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:48.039 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:48 smithi077 ceph-mon[24318]: from='osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:07:48.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:48 smithi077 ceph-mon[24318]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:07:48.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:48 smithi008 ceph-mon[21873]: from='osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:07:48.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:48 smithi008 ceph-mon[21873]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:07:48.560 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 4 on host 'smithi077' 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:07:49.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:49.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:49.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:49.166 DEBUG:teuthology.orchestra.run.smithi077:osd.4> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.4.service 2024-04-27T02:07:49.169 INFO:tasks.cephadm:Deploying osd.5 on smithi077 with /dev/vg_nvme/lv_3... 2024-04-27T02:07:49.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:07:49.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:49.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:07:50.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:50 smithi008 ceph-mon[21873]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:07:50.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:50 smithi008 ceph-mon[21873]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T02:07:50.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:50 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:50.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:50 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:50.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:50 smithi077 ceph-mon[24318]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:07:50.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:50 smithi077 ceph-mon[24318]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T02:07:50.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:50 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:50.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:50 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247] boot 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T02:07:51.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:51.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:07:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:07:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T02:07:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: osd.4 [v2:172.21.15.77:6800/1772943247,v1:172.21.15.77:6801/1772943247] boot 2024-04-27T02:07:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T02:07:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:07:52.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T02:07:52.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:52.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.137 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:52 smithi077 ceph-mon[24318]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T02:07:52.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T02:07:52.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:07:52.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:07:52.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:52 smithi008 ceph-mon[21873]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013181 s, 796 MB/s 2024-04-27T02:07:53.021 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-27T02:07:53.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:53 smithi077 ceph-mon[24318]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:53.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:53 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi077 2024-04-27T02:07:53.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:53 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi077 to 15936M 2024-04-27T02:07:53.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:53 smithi008 ceph-mon[21873]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:53.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:53 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi077 2024-04-27T02:07:53.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:53 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi077 to 15936M 2024-04-27T02:07:53.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-04-27T02:07:54.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:54 smithi077 ceph-mon[24318]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T02:07:54.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:54 smithi008 ceph-mon[21873]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T02:07:55.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:55 smithi077 ceph-mon[24318]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:55.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:55 smithi077 ceph-mon[24318]: osdmap e36: 5 total, 5 up, 5 in 2024-04-27T02:07:55.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:55 smithi008 ceph-mon[21873]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:55.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:55 smithi008 ceph-mon[21873]: osdmap e36: 5 total, 5 up, 5 in 2024-04-27T02:07:56.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:56.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:56.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:56.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:07:56.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:07:56.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:07:56.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:07:56.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:07:56.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:07:56.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:07:57.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:57 smithi008 ceph-mon[21873]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:57.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:57 smithi008 ceph-mon[21873]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:57.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:57 smithi077 ceph-mon[24318]: from='client.24165 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:07:57.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:57 smithi077 ceph-mon[24318]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:07:58.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:58 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/1374334410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]: dispatch 2024-04-27T02:07:58.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:58 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]: dispatch 2024-04-27T02:07:58.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:58 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]': finished 2024-04-27T02:07:58.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:58 smithi008 ceph-mon[21873]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T02:07:58.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:58 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:07:58.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:58 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/1374334410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]: dispatch 2024-04-27T02:07:58.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:58 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]: dispatch 2024-04-27T02:07:58.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:58 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6b059d-e641-428e-a2aa-bb6b28b319d5"}]': finished 2024-04-27T02:07:58.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:58 smithi077 ceph-mon[24318]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T02:07:58.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:58 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:07:59.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:59 smithi008 ceph-mon[21873]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:07:59.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:07:59 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2191784736' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:07:59.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:59 smithi077 ceph-mon[24318]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:07:59.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:07:59 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/2191784736' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:08:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:01 smithi008 ceph-mon[21873]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-27T02:08:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:01 smithi077 ceph-mon[24318]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-27T02:08:03.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:03 smithi077 ceph-mon[24318]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T02:08:03.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:03 smithi008 ceph-mon[21873]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T02:08:04.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:08:04.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:04.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:08:04.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:05.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:05 smithi077 ceph-mon[24318]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-27T02:08:05.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:05 smithi077 ceph-mon[24318]: Deploying daemon osd.5 on smithi077 2024-04-27T02:08:05.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:05 smithi008 ceph-mon[21873]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-27T02:08:05.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:05 smithi008 ceph-mon[21873]: Deploying daemon osd.5 on smithi077 2024-04-27T02:08:07.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:07 smithi008 ceph-mon[21873]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:08:07.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:07 smithi077 ceph-mon[24318]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:08:09.318 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:09 smithi077 ceph-mon[24318]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:09.318 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:09.318 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:09.318 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:09.318 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:09 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:09.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:09 smithi008 ceph-mon[21873]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:09.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:09.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:09.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:09.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:09 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:11.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:11 smithi008 ceph-mon[21873]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:11 smithi077 ceph-mon[24318]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:11.810 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 5 on host 'smithi077' 2024-04-27T02:08:12.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:12 smithi077 ceph-mon[24318]: from='osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:08:12.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:12 smithi077 ceph-mon[24318]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:08:12.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:12 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:12.410 DEBUG:teuthology.orchestra.run.smithi077:osd.5> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.5.service 2024-04-27T02:08:12.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:12 smithi008 ceph-mon[21873]: from='osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:08:12.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:12 smithi008 ceph-mon[21873]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:08:12.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:12 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:12.413 INFO:tasks.cephadm:Deploying osd.6 on smithi077 with /dev/vg_nvme/lv_2... 2024-04-27T02:08:12.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: from='osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:13.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:13 smithi008 ceph-mon[21873]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:13.434 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:13.434 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:08:13.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T02:08:13.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:13.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: from='osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:13.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-mon[24318]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:13.435 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[31418]: 2024-04-27T02:08:13.106+0000 7ff275c08700 -1 osd.5 0 waiting for initial osdmap 2024-04-27T02:08:13.435 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:08:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[31418]: 2024-04-27T02:08:13.112+0000 7ff270da0700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: osdmap e39: 6 total, 5 up, 6 in 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676] boot 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T02:08:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:14 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:08:14.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: osdmap e39: 6 total, 5 up, 6 in 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: osd.5 [v2:172.21.15.77:6808/3567550676,v1:172.21.15.77:6809/3567550676] boot 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T02:08:14.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:14 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150204 s, 698 MB/s 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 2024-04-27T02:08:14.799 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-27T02:08:15.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:15 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:08:15.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:15 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:08:15.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:15 smithi077 ceph-mon[24318]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:15.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:15 smithi077 ceph-mon[24318]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T02:08:15.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:15 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:08:15.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:15 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:08:15.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:15 smithi008 ceph-mon[21873]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T02:08:15.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:15 smithi008 ceph-mon[21873]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T02:08:16.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-04-27T02:08:17.117 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:16 smithi077 ceph-mon[24318]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:17.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:16 smithi077 ceph-mon[24318]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T02:08:17.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:16 smithi077 ceph-mon[24318]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:08:17.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:16 smithi008 ceph-mon[21873]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:17.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:16 smithi008 ceph-mon[21873]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T02:08:17.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:16 smithi008 ceph-mon[21873]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:08:18.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi077 2024-04-27T02:08:18.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:18.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi077 to 7968M 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:18.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:18 smithi008 ceph-mon[21873]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T02:08:18.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi077 2024-04-27T02:08:18.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi077 to 7968M 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:18.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:18 smithi077 ceph-mon[24318]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T02:08:20.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:19 smithi008 ceph-mon[21873]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T02:08:20.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:19 smithi008 ceph-mon[21873]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:08:20.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:19 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:08:20.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:19 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:08:20.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:19 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:20.172 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:19 smithi077 ceph-mon[24318]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T02:08:20.172 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:19 smithi077 ceph-mon[24318]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:08:20.172 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:19 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:08:20.172 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:19 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:08:20.172 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:19 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:21.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:20 smithi077 ceph-mon[24318]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:21.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:20 smithi077 ceph-mon[24318]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T02:08:21.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:20 smithi077 ceph-mon[24318]: Cluster is now healthy 2024-04-27T02:08:21.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:20 smithi008 ceph-mon[21873]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:21.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:20 smithi008 ceph-mon[21873]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T02:08:21.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:20 smithi008 ceph-mon[21873]: Cluster is now healthy 2024-04-27T02:08:22.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:21 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2477149216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]: dispatch 2024-04-27T02:08:22.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:21 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]: dispatch 2024-04-27T02:08:22.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:21 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]': finished 2024-04-27T02:08:22.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:21 smithi008 ceph-mon[21873]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T02:08:22.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:21 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:22.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:21 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/2477149216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]: dispatch 2024-04-27T02:08:22.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:21 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]: dispatch 2024-04-27T02:08:22.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:21 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d6f614f-c243-45c9-bd83-c7f887490831"}]': finished 2024-04-27T02:08:22.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:21 smithi077 ceph-mon[24318]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T02:08:22.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:21 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:23.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:22 smithi008 ceph-mon[21873]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-27T02:08:23.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:22 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/831684437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:08:23.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:22 smithi077 ceph-mon[24318]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-27T02:08:23.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:22 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/831684437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:08:24.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:24 smithi077 ceph-mon[24318]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:08:25.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:24 smithi008 ceph-mon[21873]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:08:27.126 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:26 smithi077 ceph-mon[24318]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-27T02:08:27.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:26 smithi008 ceph-mon[21873]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-27T02:08:28.137 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:08:28.137 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:28.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:08:28.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:28.876 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:28 smithi077 ceph-mon[24318]: Deploying daemon osd.6 on smithi077 2024-04-27T02:08:28.876 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:28 smithi077 ceph-mon[24318]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-27T02:08:29.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:28 smithi008 ceph-mon[21873]: Deploying daemon osd.6 on smithi077 2024-04-27T02:08:29.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:28 smithi008 ceph-mon[21873]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-27T02:08:31.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:30 smithi008 ceph-mon[21873]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:31.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:30 smithi077 ceph-mon[24318]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:33 smithi008 ceph-mon[21873]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:33.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:33 smithi077 ceph-mon[24318]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:35.176 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:35 smithi077 ceph-mon[24318]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:35.186 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 6 on host 'smithi077' 2024-04-27T02:08:35.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:35 smithi008 ceph-mon[21873]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:35.807 DEBUG:teuthology.orchestra.run.smithi077:osd.6> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.6.service 2024-04-27T02:08:35.810 INFO:tasks.cephadm:Deploying osd.7 on smithi077 with /dev/vg_nvme/lv_1... 2024-04-27T02:08:35.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T02:08:36.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:36 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:36.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:36 smithi077 ceph-mon[24318]: from='osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:08:36.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:36 smithi077 ceph-mon[24318]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:08:36.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:36 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:36.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:36 smithi008 ceph-mon[21873]: from='osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:08:36.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:36 smithi008 ceph-mon[21873]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:08:37.374 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:37.374 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T02:08:37.374 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T02:08:37.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:37.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: from='osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:37.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-mon[24318]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:37.375 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[34397]: 2024-04-27T02:08:37.216+0000 7f064e7a2700 -1 osd.6 0 waiting for initial osdmap 2024-04-27T02:08:37.375 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:08:37 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[34397]: 2024-04-27T02:08:37.222+0000 7f0649139700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: from='osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:37.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:37 smithi008 ceph-mon[21873]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:08:38.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:08:38.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:08:38.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:08:38.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: osdmap e47: 7 total, 6 up, 7 in 2024-04-27T02:08:38.449 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:38.449 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:38.449 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T02:08:38.562 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T02:08:38.562 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-04-27T02:08:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-04-27T02:08:38.563 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01214 s, 864 MB/s 2024-04-27T02:08:38.563 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: osdmap e47: 7 total, 6 up, 7 in 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:38.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:39 smithi077 ceph-mon[24318]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:39 smithi077 ceph-mon[24318]: osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330] boot 2024-04-27T02:08:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:39 smithi077 ceph-mon[24318]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T02:08:39.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:39 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:39.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:39 smithi008 ceph-mon[21873]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T02:08:39.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:39 smithi008 ceph-mon[21873]: osd.6 [v2:172.21.15.77:6816/3822818330,v1:172.21.15.77:6817/3822818330] boot 2024-04-27T02:08:39.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:39 smithi008 ceph-mon[21873]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T02:08:39.561 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:39 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:08:40.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-04-27T02:08:40.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:40 smithi077 ceph-mon[24318]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T02:08:40.585 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:40 smithi008 ceph-mon[21873]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T02:08:41.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:41 smithi008 ceph-mon[21873]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:41.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:41 smithi008 ceph-mon[21873]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T02:08:41.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:41 smithi077 ceph-mon[24318]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:41.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:41 smithi077 ceph-mon[24318]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T02:08:42.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:42.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:42.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:42.661 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi077 2024-04-27T02:08:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi077 to 5312M 2024-04-27T02:08:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: osdmap e52: 7 total, 7 up, 7 in 2024-04-27T02:08:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:08:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:08:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi077 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi077 to 5312M 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: osdmap e52: 7 total, 7 up, 7 in 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T02:08:43.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:44.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:44 smithi077 ceph-mon[24318]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:08:44.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:44 smithi008 ceph-mon[21873]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:08:45.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:45.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/2716439916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]: dispatch 2024-04-27T02:08:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]: dispatch 2024-04-27T02:08:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]': finished 2024-04-27T02:08:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T02:08:45.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:45 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/2716439916' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]: dispatch 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]: dispatch 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e41953d7-a787-462f-9bc9-703e84f55af5"}]': finished 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T02:08:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:45 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:08:46.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:46 smithi077 ceph-mon[24318]: from='client.? 172.21.15.77:0/1691488305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:08:46.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:46 smithi008 ceph-mon[21873]: from='client.? 172.21.15.77:0/1691488305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T02:08:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:47 smithi077 ceph-mon[24318]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-27T02:08:47.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:47 smithi008 ceph-mon[21873]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-27T02:08:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:49 smithi077 ceph-mon[24318]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-27T02:08:49.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:49 smithi008 ceph-mon[21873]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-27T02:08:51.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:51 smithi077 ceph-mon[24318]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T02:08:51.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:08:51.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:51.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:51 smithi008 ceph-mon[21873]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T02:08:51.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:08:51.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:52.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:52 smithi077 ceph-mon[24318]: Deploying daemon osd.7 on smithi077 2024-04-27T02:08:52.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:52 smithi008 ceph-mon[21873]: Deploying daemon osd.7 on smithi077 2024-04-27T02:08:53.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:53 smithi077 ceph-mon[24318]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-27T02:08:53.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:53 smithi008 ceph-mon[21873]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-27T02:08:54.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:54 smithi077 ceph-mon[24318]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:08:54.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:54 smithi008 ceph-mon[21873]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:08:56.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:08:56.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:08:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:08:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:08:57.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:56 smithi008 ceph-mon[21873]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:57.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:57.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:57.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:57.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:57.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:56 smithi077 ceph-mon[24318]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:57.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:08:57.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:08:57.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:08:57.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:08:59.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:58 smithi008 ceph-mon[21873]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:59.207 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:58 smithi077 ceph-mon[24318]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:08:59.239 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 7 on host 'smithi077' 2024-04-27T02:09:00.141 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:59 smithi077 ceph-mon[24318]: from='osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:09:00.141 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:59 smithi077 ceph-mon[24318]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:09:00.141 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:08:59 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:00.142 DEBUG:teuthology.orchestra.run.smithi077:osd.7> sudo journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.7.service 2024-04-27T02:09:00.145 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-27T02:09:00.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T02:09:00.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:59 smithi008 ceph-mon[21873]: from='osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:09:00.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:59 smithi008 ceph-mon[21873]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:09:00.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:08:59 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: from='osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:09:01.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:00 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:09:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T02:09:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T02:09:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: from='osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:09:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:09:01.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:01.236 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[37493]: 2024-04-27T02:09:00.933+0000 7fafafc57700 -1 osd.7 0 waiting for initial osdmap 2024-04-27T02:09:01.236 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:09:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[37493]: 2024-04-27T02:09:00.941+0000 7fafaadef700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:09:02.135 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:02.135 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:01 smithi008 ceph-mon[21873]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:09:02.135 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:01 smithi008 ceph-mon[21873]: osdmap e55: 8 total, 7 up, 8 in 2024-04-27T02:09:02.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:02.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:01 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:02.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:01 smithi077 ceph-mon[24318]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-04-27T02:09:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:01 smithi077 ceph-mon[24318]: osdmap e55: 8 total, 7 up, 8 in 2024-04-27T02:09:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:02.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:01 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:02.822 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1714183741,"num_in_osds":8,"osd_in_since":1714183724,"num_remapped_pgs":1} 2024-04-27T02:09:02.823 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:09:02.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: purged_snaps scrub starts 2024-04-27T02:09:02.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: purged_snaps scrub ok 2024-04-27T02:09:02.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:09:02.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702] boot 2024-04-27T02:09:02.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T02:09:02.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:02.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:02 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3072666943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:09:03.092 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:03.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: purged_snaps scrub starts 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: purged_snaps scrub ok 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: osd.7 [v2:172.21.15.77:6824/3400207702,v1:172.21.15.77:6825/3400207702] boot 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:09:03.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:02 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3072666943' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T02:09:04.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: Detected new or changed devices on smithi077 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi077 to 3984M 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:09:04.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:03 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.272 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: Detected new or changed devices on smithi077 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi077 to 3984M 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:09:04.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:03 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:09:04.960 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:04.960 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":59,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","created":"2024-04-27T02:03:02.518543+0000","modified":"2024-04-27T02:09:04.865336+0000","last_up_change":"2024-04-27T02:09:01.931385+0000","last_in_change":"2024-04-27T02:08:44.810938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:07:04.038387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"426fa516-32ce-4ac8-90f6-fb5cf5ed20ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6803","nonce":425614476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6805","nonce":425614476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6809","nonce":425614476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6807","nonce":425614476}]},"public_addr":"172.21.15.8:6803/425614476","cluster_addr":"172.21.15.8:6805/425614476","heartbeat_back_addr":"172.21.15.8:6809/425614476","heartbeat_front_addr":"172.21.15.8:6807/425614476","state":["exists","up"]},{"osd":1,"uuid":"3f83e4d6-fbf7-449b-aaef-f60f5a68641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6811","nonce":2674833045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6813","nonce":2674833045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6817","nonce":2674833045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6815","nonce":2674833045}]},"public_addr":"172.21.15.8:6811/2674833045","cluster_addr":"172.21.15.8:6813/2674833045","heartbeat_back_addr":"172.21.15.8:6817/2674833045","heartbeat_front_addr":"172.21.15.8:6815/2674833045","state":["exists","up"]},{"osd":2,"uuid":"b8081e46-868d-4256-8838-462607a616b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6819","nonce":896710746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6821","nonce":896710746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6825","nonce":896710746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6823","nonce":896710746}]},"public_addr":"172.21.15.8:6819/896710746","cluster_addr":"172.21.15.8:6821/896710746","heartbeat_back_addr":"172.21.15.8:6825/896710746","heartbeat_front_addr":"172.21.15.8:6823/896710746","state":["exists","up"]},{"osd":3,"uuid":"32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6827","nonce":3468489236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6829","nonce":3468489236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6833","nonce":3468489236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6831","nonce":3468489236}]},"public_addr":"172.21.15.8:6827/3468489236","cluster_addr":"172.21.15.8:6829/3468489236","heartbeat_back_addr":"172.21.15.8:6833/3468489236","heartbeat_front_addr":"172.21.15.8:6831/3468489236","state":["exists","up"]},{"osd":4,"uuid":"248d0710-cd99-4824-86c1-f1b93e727cb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6801","nonce":1772943247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6803","nonce":1772943247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6807","nonce":1772943247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6805","nonce":1772943247}]},"public_addr":"172.21.15.77:6801/1772943247","cluster_addr":"172.21.15.77:6803/1772943247","heartbeat_back_addr":"172.21.15.77:6807/1772943247","heartbeat_front_addr":"172.21.15.77:6805/1772943247","state":["exists","up"]},{"osd":5,"uuid":"1e6b059d-e641-428e-a2aa-bb6b28b319d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6809","nonce":3567550676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6811","nonce":3567550676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6815","nonce":3567550676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6813","nonce":3567550676}]},"public_addr":"172.21.15.77:6809/3567550676","cluster_addr":"172.21.15.77:6811/3567550676","heartbeat_back_addr":"172.21.15.77:6815/3567550676","heartbeat_front_addr":"172.21.15.77:6813/3567550676","state":["exists","up"]},{"osd":6,"uuid":"8d6f614f-c243-45c9-bd83-c7f887490831","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6817","nonce":3822818330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6819","nonce":3822818330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6823","nonce":3822818330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6821","nonce":3822818330}]},"public_addr":"172.21.15.77:6817/3822818330","cluster_addr":"172.21.15.77:6819/3822818330","heartbeat_back_addr":"172.21.15.77:6823/3822818330","heartbeat_front_addr":"172.21.15.77:6821/3822818330","state":["exists","up"]},{"osd":7,"uuid":"e41953d7-a787-462f-9bc9-703e84f55af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6825","nonce":3400207702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6827","nonce":3400207702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6831","nonce":3400207702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6829","nonce":3400207702}]},"public_addr":"172.21.15.77:6825/3400207702","cluster_addr":"172.21.15.77:6827/3400207702","heartbeat_back_addr":"172.21.15.77:6831/3400207702","heartbeat_front_addr":"172.21.15.77:6829/3400207702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:03.069607+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:34.163627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:00.089716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:26.519517+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:48.924831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:12.860259+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:36.205080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:09:00.014757+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:6800/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/2033385972":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/3466161845":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1365123807":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/4227358918":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6801/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6800/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:6800/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/271097614":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1715551387":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/2897716044":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/1175135893":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/2790303437":"2024-04-28T02:03:30.980116+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:09:05.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:04 smithi077 ceph-mon[24318]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:05.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:04 smithi077 ceph-mon[24318]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T02:09:05.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:04 smithi077 ceph-mon[24318]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T02:09:05.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:04 smithi008 ceph-mon[21873]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:05.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:04 smithi008 ceph-mon[21873]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T02:09:05.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:04 smithi008 ceph-mon[21873]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T02:09:05.989 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T02:07:04.038387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-27T02:09:05.989 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-27T02:09:06.219 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:06.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:05 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2958671934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:06.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:05 smithi077 ceph-mon[24318]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T02:09:06.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:05 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2958671934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:06.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:05 smithi008 ceph-mon[21873]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T02:09:07.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:06 smithi077 ceph-mon[24318]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:07.343 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:06 smithi008 ceph-mon[21873]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:08.184 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 1 2024-04-27T02:09:08.907 INFO:tasks.cephadm:Setting up client nodes... 2024-04-27T02:09:08.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T02:09:09.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:08 smithi008 ceph-mon[21873]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:09.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:08 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1582322220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:09:09.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:08 smithi077 ceph-mon[24318]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:09.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:08 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1582322220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T02:09:10.963 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-04-27T02:09:10.963 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQBGXixmCS58OhAAHgNs3c8+AJJwc+L3ne1qTg== 2024-04-27T02:09:11.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:10 smithi077 ceph-mon[24318]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:09:11.309 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:10 smithi008 ceph-mon[21873]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T02:09:11.704 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-04-27T02:09:11.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-27T02:09:11.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T02:09:11.743 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T02:09:11.743 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-27T02:09:11.743 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-27T02:09:12.005 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:12.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:11 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2917216772' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:09:12.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:11 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2917216772' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:09:12.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:11 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2917216772' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T02:09:12.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:11 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2917216772' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T02:09:13.233 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:12 smithi008 ceph-mon[21873]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-27T02:09:13.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:12 smithi077 ceph-mon[24318]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-27T02:09:13.710 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:14.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:13 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3754576211' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T02:09:14.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:13 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3754576211' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T02:09:14.458 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi008.exkhsc","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":1484809748},{"type":"v1","addr":"172.21.15.8:6801","nonce":1484809748}]},"active_addr":"172.21.15.8:6801/1484809748","active_change":"2024-04-27T02:04:55.484792+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14228,"name":"smithi077.snksrp","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.8:8443/","prometheus":"http://172.21.15.8:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2043524281}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":4209801878}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3008185708}]},{"addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":4041653950}]}]}} 2024-04-27T02:09:14.463 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-27T02:09:14.463 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-27T02:09:14.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:09:14.686 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:15.151 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:14 smithi008 ceph-mon[21873]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-27T02:09:15.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:14 smithi077 ceph-mon[24318]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-27T02:09:16.535 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:16.535 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","created":"2024-04-27T02:03:02.518543+0000","modified":"2024-04-27T02:09:05.870513+0000","last_up_change":"2024-04-27T02:09:01.931385+0000","last_in_change":"2024-04-27T02:08:44.810938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:07:04.038387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"426fa516-32ce-4ac8-90f6-fb5cf5ed20ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6803","nonce":425614476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6805","nonce":425614476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6809","nonce":425614476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6807","nonce":425614476}]},"public_addr":"172.21.15.8:6803/425614476","cluster_addr":"172.21.15.8:6805/425614476","heartbeat_back_addr":"172.21.15.8:6809/425614476","heartbeat_front_addr":"172.21.15.8:6807/425614476","state":["exists","up"]},{"osd":1,"uuid":"3f83e4d6-fbf7-449b-aaef-f60f5a68641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6811","nonce":2674833045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6813","nonce":2674833045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6817","nonce":2674833045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6815","nonce":2674833045}]},"public_addr":"172.21.15.8:6811/2674833045","cluster_addr":"172.21.15.8:6813/2674833045","heartbeat_back_addr":"172.21.15.8:6817/2674833045","heartbeat_front_addr":"172.21.15.8:6815/2674833045","state":["exists","up"]},{"osd":2,"uuid":"b8081e46-868d-4256-8838-462607a616b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6819","nonce":896710746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6821","nonce":896710746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6825","nonce":896710746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6823","nonce":896710746}]},"public_addr":"172.21.15.8:6819/896710746","cluster_addr":"172.21.15.8:6821/896710746","heartbeat_back_addr":"172.21.15.8:6825/896710746","heartbeat_front_addr":"172.21.15.8:6823/896710746","state":["exists","up"]},{"osd":3,"uuid":"32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6827","nonce":3468489236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6829","nonce":3468489236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6833","nonce":3468489236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6831","nonce":3468489236}]},"public_addr":"172.21.15.8:6827/3468489236","cluster_addr":"172.21.15.8:6829/3468489236","heartbeat_back_addr":"172.21.15.8:6833/3468489236","heartbeat_front_addr":"172.21.15.8:6831/3468489236","state":["exists","up"]},{"osd":4,"uuid":"248d0710-cd99-4824-86c1-f1b93e727cb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6801","nonce":1772943247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6803","nonce":1772943247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6807","nonce":1772943247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6805","nonce":1772943247}]},"public_addr":"172.21.15.77:6801/1772943247","cluster_addr":"172.21.15.77:6803/1772943247","heartbeat_back_addr":"172.21.15.77:6807/1772943247","heartbeat_front_addr":"172.21.15.77:6805/1772943247","state":["exists","up"]},{"osd":5,"uuid":"1e6b059d-e641-428e-a2aa-bb6b28b319d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6809","nonce":3567550676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6811","nonce":3567550676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6815","nonce":3567550676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6813","nonce":3567550676}]},"public_addr":"172.21.15.77:6809/3567550676","cluster_addr":"172.21.15.77:6811/3567550676","heartbeat_back_addr":"172.21.15.77:6815/3567550676","heartbeat_front_addr":"172.21.15.77:6813/3567550676","state":["exists","up"]},{"osd":6,"uuid":"8d6f614f-c243-45c9-bd83-c7f887490831","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6817","nonce":3822818330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6819","nonce":3822818330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6823","nonce":3822818330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6821","nonce":3822818330}]},"public_addr":"172.21.15.77:6817/3822818330","cluster_addr":"172.21.15.77:6819/3822818330","heartbeat_back_addr":"172.21.15.77:6823/3822818330","heartbeat_front_addr":"172.21.15.77:6821/3822818330","state":["exists","up"]},{"osd":7,"uuid":"e41953d7-a787-462f-9bc9-703e84f55af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6825","nonce":3400207702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6827","nonce":3400207702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6831","nonce":3400207702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6829","nonce":3400207702}]},"public_addr":"172.21.15.77:6825/3400207702","cluster_addr":"172.21.15.77:6827/3400207702","heartbeat_back_addr":"172.21.15.77:6831/3400207702","heartbeat_front_addr":"172.21.15.77:6829/3400207702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:03.069607+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:34.163627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:00.089716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:26.519517+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:48.924831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:12.860259+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:36.205080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:09:00.014757+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:6800/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/2033385972":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/3466161845":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1365123807":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/4227358918":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6801/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6800/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:6800/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/271097614":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1715551387":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/2897716044":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/1175135893":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/2790303437":"2024-04-28T02:03:30.980116+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:09:17.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:16 smithi077 ceph-mon[24318]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:09:17.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:16 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2019758392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:17.262 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:16 smithi008 ceph-mon[21873]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T02:09:17.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:16 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2019758392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:17.294 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-27T02:09:17.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T02:09:17.521 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:19.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:18 smithi077 ceph-mon[24318]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-27T02:09:19.338 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:19.338 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":60,"fsid":"27e8cf32-043a-11ef-bc93-c7b262605968","created":"2024-04-27T02:03:02.518543+0000","modified":"2024-04-27T02:09:05.870513+0000","last_up_change":"2024-04-27T02:09:01.931385+0000","last_in_change":"2024-04-27T02:08:44.810938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T02:07:04.038387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"426fa516-32ce-4ac8-90f6-fb5cf5ed20ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6803","nonce":425614476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6805","nonce":425614476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6809","nonce":425614476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":425614476},{"type":"v1","addr":"172.21.15.8:6807","nonce":425614476}]},"public_addr":"172.21.15.8:6803/425614476","cluster_addr":"172.21.15.8:6805/425614476","heartbeat_back_addr":"172.21.15.8:6809/425614476","heartbeat_front_addr":"172.21.15.8:6807/425614476","state":["exists","up"]},{"osd":1,"uuid":"3f83e4d6-fbf7-449b-aaef-f60f5a68641e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6811","nonce":2674833045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6813","nonce":2674833045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6817","nonce":2674833045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":2674833045},{"type":"v1","addr":"172.21.15.8:6815","nonce":2674833045}]},"public_addr":"172.21.15.8:6811/2674833045","cluster_addr":"172.21.15.8:6813/2674833045","heartbeat_back_addr":"172.21.15.8:6817/2674833045","heartbeat_front_addr":"172.21.15.8:6815/2674833045","state":["exists","up"]},{"osd":2,"uuid":"b8081e46-868d-4256-8838-462607a616b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6819","nonce":896710746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6821","nonce":896710746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6825","nonce":896710746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":896710746},{"type":"v1","addr":"172.21.15.8:6823","nonce":896710746}]},"public_addr":"172.21.15.8:6819/896710746","cluster_addr":"172.21.15.8:6821/896710746","heartbeat_back_addr":"172.21.15.8:6825/896710746","heartbeat_front_addr":"172.21.15.8:6823/896710746","state":["exists","up"]},{"osd":3,"uuid":"32c22e81-7e4d-483b-ae1f-1de3f0b9ed4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6827","nonce":3468489236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6829","nonce":3468489236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6833","nonce":3468489236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3468489236},{"type":"v1","addr":"172.21.15.8:6831","nonce":3468489236}]},"public_addr":"172.21.15.8:6827/3468489236","cluster_addr":"172.21.15.8:6829/3468489236","heartbeat_back_addr":"172.21.15.8:6833/3468489236","heartbeat_front_addr":"172.21.15.8:6831/3468489236","state":["exists","up"]},{"osd":4,"uuid":"248d0710-cd99-4824-86c1-f1b93e727cb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6801","nonce":1772943247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6803","nonce":1772943247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6807","nonce":1772943247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1772943247},{"type":"v1","addr":"172.21.15.77:6805","nonce":1772943247}]},"public_addr":"172.21.15.77:6801/1772943247","cluster_addr":"172.21.15.77:6803/1772943247","heartbeat_back_addr":"172.21.15.77:6807/1772943247","heartbeat_front_addr":"172.21.15.77:6805/1772943247","state":["exists","up"]},{"osd":5,"uuid":"1e6b059d-e641-428e-a2aa-bb6b28b319d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6809","nonce":3567550676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6811","nonce":3567550676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6815","nonce":3567550676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3567550676},{"type":"v1","addr":"172.21.15.77:6813","nonce":3567550676}]},"public_addr":"172.21.15.77:6809/3567550676","cluster_addr":"172.21.15.77:6811/3567550676","heartbeat_back_addr":"172.21.15.77:6815/3567550676","heartbeat_front_addr":"172.21.15.77:6813/3567550676","state":["exists","up"]},{"osd":6,"uuid":"8d6f614f-c243-45c9-bd83-c7f887490831","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6817","nonce":3822818330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6819","nonce":3822818330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6823","nonce":3822818330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":3822818330},{"type":"v1","addr":"172.21.15.77:6821","nonce":3822818330}]},"public_addr":"172.21.15.77:6817/3822818330","cluster_addr":"172.21.15.77:6819/3822818330","heartbeat_back_addr":"172.21.15.77:6823/3822818330","heartbeat_front_addr":"172.21.15.77:6821/3822818330","state":["exists","up"]},{"osd":7,"uuid":"e41953d7-a787-462f-9bc9-703e84f55af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6825","nonce":3400207702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6827","nonce":3400207702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6831","nonce":3400207702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3400207702},{"type":"v1","addr":"172.21.15.77:6829","nonce":3400207702}]},"public_addr":"172.21.15.77:6825/3400207702","cluster_addr":"172.21.15.77:6827/3400207702","heartbeat_back_addr":"172.21.15.77:6831/3400207702","heartbeat_front_addr":"172.21.15.77:6829/3400207702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:03.069607+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:06:34.163627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:00.089716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:26.519517+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:07:48.924831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:12.860259+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:08:36.205080+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T02:09:00.014757+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:6800/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/2033385972":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/3466161845":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1365123807":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/4227358918":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6801/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/888741760":"2024-04-28T02:04:55.484511+0000","172.21.15.8:6800/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:6800/1351148643":"2024-04-28T02:03:59.990182+0000","172.21.15.8:6801/2053103416":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/271097614":"2024-04-28T02:04:55.484511+0000","172.21.15.8:0/1715551387":"2024-04-28T02:03:59.990182+0000","172.21.15.8:0/2897716044":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/1175135893":"2024-04-28T02:03:30.980116+0000","172.21.15.8:0/2790303437":"2024-04-28T02:03:30.980116+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T02:09:19.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:18 smithi008 ceph-mon[21873]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-27T02:09:19.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-27T02:09:19.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-27T02:09:19.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-27T02:09:19.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-27T02:09:19.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-27T02:09:19.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-27T02:09:19.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-27T02:09:19.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-27T02:09:20.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:19 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1452560089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:20.396 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.396 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.397 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.397 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.401 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.403 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:19 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1452560089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T02:09:20.432 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:20.433 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:21.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:21 smithi008 ceph-mon[21873]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-27T02:09:21.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:21 smithi077 ceph-mon[24318]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-27T02:09:23.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:23 smithi008 ceph-mon[21873]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:23.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:23 smithi077 ceph-mon[24318]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:25.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:25 smithi008 ceph-mon[21873]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:25.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:25 smithi077 ceph-mon[24318]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:27.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:27 smithi008 ceph-mon[21873]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:27.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:27 smithi077 ceph-mon[24318]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:29.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:29 smithi008 ceph-mon[21873]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:29.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:29 smithi077 ceph-mon[24318]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:30.312 INFO:teuthology.orchestra.run.smithi008.stdout:115964117019 2024-04-27T02:09:30.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T02:09:31.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:31 smithi008 ceph-mon[21873]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:31.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:31 smithi077 ceph-mon[24318]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:31.559 INFO:teuthology.orchestra.run.smithi008.stdout:137438953494 2024-04-27T02:09:31.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T02:09:33.032 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:33 smithi008 ceph-mon[21873]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:33.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:33 smithi077 ceph-mon[24318]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:34.531 INFO:teuthology.orchestra.run.smithi008.stdout:171798691858 2024-04-27T02:09:34.531 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T02:09:34.983 INFO:teuthology.orchestra.run.smithi008.stdout:81604378657 2024-04-27T02:09:34.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T02:09:35.059 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:35.083 INFO:teuthology.orchestra.run.smithi008.stdout:60129542181 2024-04-27T02:09:35.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T02:09:35.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:35 smithi008 ceph-mon[21873]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:35.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:35 smithi077 ceph-mon[24318]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:36.800 INFO:teuthology.orchestra.run.smithi008.stdout:240518168585 2024-04-27T02:09:36.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-27T02:09:37.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:37 smithi008 ceph-mon[21873]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:37.470 INFO:teuthology.orchestra.run.smithi008.stdout:206158430222 2024-04-27T02:09:37.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-27T02:09:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:37 smithi077 ceph-mon[24318]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:37.662 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:37.716 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:38.661 INFO:teuthology.orchestra.run.smithi008.stdout:38654705708 2024-04-27T02:09:38.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T02:09:39.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:39 smithi008 ceph-mon[21873]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:39 smithi077 ceph-mon[24318]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:39.496 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:39.727 INFO:teuthology.orchestra.run.smithi008.stdout:115964117020 2024-04-27T02:09:40.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:40 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/619093547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T02:09:40.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:40 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/619093547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T02:09:41.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:41 smithi008 ceph-mon[21873]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:41.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:41 smithi077 ceph-mon[24318]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:41.570 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:42.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117019 got 115964117020 for osd.3 2024-04-27T02:09:42.002 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:43.382 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:43 smithi008 ceph-mon[21873]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:43 smithi077 ceph-mon[24318]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:43.794 INFO:teuthology.orchestra.run.smithi008.stdout:81604378659 2024-04-27T02:09:44.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:44 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3110720762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T02:09:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:44 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3110720762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T02:09:44.528 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378659 for osd.2 2024-04-27T02:09:44.528 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:45.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:45 smithi008 ceph-mon[21873]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:45.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:45 smithi077 ceph-mon[24318]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:46.218 INFO:teuthology.orchestra.run.smithi008.stdout:60129542184 2024-04-27T02:09:46.431 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:47.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:47 smithi008 ceph-mon[21873]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:47.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:47 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3253570110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T02:09:47.300 INFO:teuthology.orchestra.run.smithi008.stdout:137438953497 2024-04-27T02:09:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:47 smithi077 ceph-mon[24318]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:47 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3253570110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T02:09:47.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953494 got 137438953497 for osd.4 2024-04-27T02:09:47.933 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:48.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:48 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1130219186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T02:09:48.345 INFO:teuthology.orchestra.run.smithi008.stdout:206158430225 2024-04-27T02:09:48.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:48 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1130219186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T02:09:48.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542181 got 60129542184 for osd.1 2024-04-27T02:09:48.725 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:49.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:49 smithi008 ceph-mon[21873]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:49.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:49 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3855232881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T02:09:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:49 smithi077 ceph-mon[24318]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:49 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3855232881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T02:09:50.173 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430225 for osd.6 2024-04-27T02:09:50.173 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:50.784 INFO:teuthology.orchestra.run.smithi008.stdout:240518168588 2024-04-27T02:09:51.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:51 smithi008 ceph-mon[21873]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:51.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:51 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/4064954471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T02:09:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:51 smithi077 ceph-mon[24318]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:51 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/4064954471' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T02:09:51.735 INFO:teuthology.orchestra.run.smithi008.stdout:171798691862 2024-04-27T02:09:51.986 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168585 got 240518168588 for osd.7 2024-04-27T02:09:51.986 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:52.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:52 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1033811324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T02:09:52.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:52 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1033811324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T02:09:52.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691858 got 171798691862 for osd.5 2024-04-27T02:09:52.580 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:53.316 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:53 smithi008 ceph-mon[21873]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:53.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:53 smithi077 ceph-mon[24318]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:53.741 INFO:teuthology.orchestra.run.smithi008.stdout:38654705711 2024-04-27T02:09:54.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:54 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2418231327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T02:09:54.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705711 for osd.0 2024-04-27T02:09:54.351 DEBUG:teuthology.parallel:result is None 2024-04-27T02:09:54.351 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T02:09:54.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T02:09:54.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:54 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2418231327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T02:09:54.574 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:55.374 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:55 smithi008 ceph-mon[21873]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:55.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:55 smithi077 ceph-mon[24318]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:56.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:09:56.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:09:56.284 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:56.286 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-27T02:09:56.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:09:56.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:09:56.978 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-27T02:09:55.591095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48868,"kb_used_data":6372,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912348,"statfs":{"total":767960285184,"available":767910244352,"internally_reserved":0,"allocated":6524928,"data_stored":3571105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001981"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-27T02:09:05.879092+0000","last_change":"2024-04-27T02:09:05.879092+0000","last_active":"2024-04-27T02:09:05.879092+0000","last_peered":"2024-04-27T02:09:05.879092+0000","last_clean":"2024-04-27T02:09:05.879092+0000","last_became_active":"2024-04-27T02:09:05.878648+0000","last_became_peered":"2024-04-27T02:09:05.878648+0000","last_unstale":"2024-04-27T02:09:05.879092+0000","last_undegraded":"2024-04-27T02:09:05.879092+0000","last_fullsized":"2024-04-27T02:09:05.879092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T02:07:04.088549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T02:07:04.088549+0000","last_clean_scrub_stamp":"2024-04-27T02:07:04.088549+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:09.532894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":776554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":1146880,"data_stored":776877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.435}]},{"osd":1,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.505}]},{"osd":5,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.35899999999999999,"15min":0.32100000000000001},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.378,"15min":0.33000000000000002},"min":{"1min":0.30599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Apr 27 02:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40799999999999997,"15min":0.40100000000000002},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.309},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.374,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64400000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Apr 27 02:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.504,"15min":0.497},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Apr 27 02:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.46400000000000002}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.44},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Apr 27 02:09:41 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 02:09:46 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.36699999999999999,"15min":0.34799999999999998},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.377},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.26200000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.629,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.314}]},{"osd":2,"last update":"Sat Apr 27 02:09:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.38700000000000001,"15min":0.376},"min":{"1min":0.29899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.33800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Apr 27 02:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Apr 27 02:09:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":0.80200000000000005,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.998999999999999,"5min":18.998999999999999,"15min":18.998999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84199999999999997,"5min":1.024,"15min":1.024},"last":0.67000000000000004}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":19.024000000000001,"5min":19.024000000000001,"15min":19.024000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.308,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Apr 27 02:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.308,"5min":0.247,"15min":0.247},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.41399999999999998,"15min":0.40799999999999997},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.502}]},{"osd":3,"last update":"Sat Apr 27 02:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.36299999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.624,"5min":0.629,"15min":0.629},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Apr 27 02:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.66600000000000004}]},{"osd":5,"last update":"Sat Apr 27 02:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.82199999999999995}]},{"osd":6,"last update":"Sat Apr 27 02:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.40999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.48499999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.311}]},{"osd":4,"last update":"Sat Apr 27 02:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.503}]},{"osd":6,"last update":"Sat Apr 27 02:09:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67600000000000005}]},{"osd":1,"last update":"Sat Apr 27 02:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Apr 27 02:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53900000000000003}]},{"osd":3,"last update":"Sat Apr 27 02:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.623}]},{"osd":6,"last update":"Sat Apr 27 02:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.59599999999999997}]},{"osd":2,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Apr 27 02:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T02:09:56.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T02:09:57.206 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:09:57.328 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:57 smithi008 ceph-mon[21873]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:57.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:57 smithi077 ceph-mon[24318]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:58.360 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:58 smithi008 ceph-mon[21873]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:09:58.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:58 smithi077 ceph-mon[24318]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:09:58.995 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:09:58.997 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-04-27T02:09:59.360 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:09:59 smithi008 ceph-mon[21873]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:59.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:09:59 smithi077 ceph-mon[24318]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:09:59.570 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-27T02:09:57.591455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48868,"kb_used_data":6372,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912348,"statfs":{"total":767960285184,"available":767910244352,"internally_reserved":0,"allocated":6524928,"data_stored":3571105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002023"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-27T02:09:05.879092+0000","last_change":"2024-04-27T02:09:05.879092+0000","last_active":"2024-04-27T02:09:05.879092+0000","last_peered":"2024-04-27T02:09:05.879092+0000","last_clean":"2024-04-27T02:09:05.879092+0000","last_became_active":"2024-04-27T02:09:05.878648+0000","last_became_peered":"2024-04-27T02:09:05.878648+0000","last_unstale":"2024-04-27T02:09:05.879092+0000","last_undegraded":"2024-04-27T02:09:05.879092+0000","last_fullsized":"2024-04-27T02:09:05.879092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T02:07:04.088549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T02:07:04.088549+0000","last_clean_scrub_stamp":"2024-04-27T02:07:04.088549+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:09.532894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":776554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":1146880,"data_stored":776877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.435}]},{"osd":1,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.505}]},{"osd":5,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.35899999999999999,"15min":0.32100000000000001},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.378,"15min":0.33000000000000002},"min":{"1min":0.30599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Apr 27 02:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40799999999999997,"15min":0.40100000000000002},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.374,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64400000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 27 02:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.504,"15min":0.497},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Apr 27 02:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.64700000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Apr 27 02:09:41 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 02:09:46 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.36699999999999999,"15min":0.34799999999999998},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.377},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.26200000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.629,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.314}]},{"osd":2,"last update":"Sat Apr 27 02:09:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.38700000000000001,"15min":0.376},"min":{"1min":0.29899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.33800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Apr 27 02:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.45000000000000001}]},{"osd":4,"last update":"Sat Apr 27 02:09:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":0.80200000000000005,"15min":0.67500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":18.998999999999999,"5min":18.998999999999999,"15min":18.998999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84199999999999997,"5min":1.024,"15min":1.024},"last":0.67000000000000004}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.355,"15min":1.355},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":19.024000000000001,"5min":19.024000000000001,"15min":19.024000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Apr 27 02:09:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.308,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.377,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Apr 27 02:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.435,"15min":0.434},"min":{"1min":0.308,"5min":0.247,"15min":0.247},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.41399999999999998,"15min":0.40799999999999997},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.502}]},{"osd":3,"last update":"Sat Apr 27 02:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.36299999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.624,"5min":0.629,"15min":0.629},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Apr 27 02:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.66600000000000004}]},{"osd":5,"last update":"Sat Apr 27 02:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.82199999999999995}]},{"osd":6,"last update":"Sat Apr 27 02:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.71499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]}]},{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.40999999999999998,"15min":0.41199999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58299999999999996,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.48499999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65800000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Apr 27 02:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.48699999999999999,"15min":0.49199999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.311}]},{"osd":4,"last update":"Sat Apr 27 02:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.503}]},{"osd":6,"last update":"Sat Apr 27 02:09:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.495}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.57199999999999995,"15min":0.58099999999999996},"min":{"1min":0.39600000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64500000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76100000000000001,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Apr 27 02:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68300000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70299999999999996,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Apr 27 02:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72999999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70199999999999996,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.496}]},{"osd":3,"last update":"Sat Apr 27 02:09:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.58999999999999997,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74099999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.442},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.59899999999999998,"15min":0.60199999999999998},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.72599999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.372}]},{"osd":6,"last update":"Sat Apr 27 02:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.59599999999999997}]},{"osd":2,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Apr 27 02:09:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Apr 27 02:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T02:09:59.572 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T02:09:59.572 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T02:09:59.572 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T02:09:59.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T02:09:59.792 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/mon.smithi008/config 2024-04-27T02:10:00.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:00 smithi008 ceph-mon[21873]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:00.173 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:00 smithi008 ceph-mon[21873]: overall HEALTH_OK 2024-04-27T02:10:00.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:00 smithi077 ceph-mon[24318]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:00.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:00 smithi077 ceph-mon[24318]: overall HEALTH_OK 2024-04-27T02:10:01.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:01 smithi008 ceph-mon[21873]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:01 smithi077 ceph-mon[24318]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:01.640 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:10:01.640 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T02:10:02.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:02 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2432940471' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T02:10:02.318 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T02:10:02.319 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T02:10:02.319 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:10:02.329 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:02.329 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-27T02:10:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:02 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2432940471' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T02:10:03.318 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:03 smithi008 ceph-mon[21873]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:03.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:03 smithi077 ceph-mon[24318]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:04.345 INFO:teuthology.orchestra.run.smithi008.stdout:Backend: cephadm 2024-04-27T02:10:04.345 INFO:teuthology.orchestra.run.smithi008.stdout:Available: Yes 2024-04-27T02:10:04.345 INFO:teuthology.orchestra.run.smithi008.stdout:Paused: No 2024-04-27T02:10:04.345 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:04.345 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:04.345 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:04.345 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:04 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:04.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:04.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:04.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:04.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:04 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:04.968 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-27T02:10:05.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:05 smithi008 ceph-mon[21873]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:05.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:05 smithi077 ceph-mon[24318]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:06.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:06 smithi008 ceph-mon[21873]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:06.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:06 smithi077 ceph-mon[24318]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:07.011 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:10:07.011 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c aa5e9072673c 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 5m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (4m) 66s ago 4m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (4m) 2m ago 5m 44.4M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (4m) 66s ago 4m 412M - 17.2.0 e1d6a67b021e 1c7523763953 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 2m ago 7m 42.6M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (4m) 66s ago 4m 38.5M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 2m ago 5m 17.7M - 1dbe0e931976 3563103131a3 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (4m) 66s ago 4m 17.6M - 1dbe0e931976 d9d76f2322b0 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 2m ago 4m 31.8M 3217M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 2m ago 3m 34.7M 3217M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 3m 30.0M 3217M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:10:07.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 2m 10.4M 3217M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:10:07.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (2m) 66s ago 2m 35.7M 3984M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:10:07.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (118s) 66s ago 118s 32.3M 3984M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:10:07.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (95s) 66s ago 94s 30.4M 3984M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:10:07.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (71s) 66s ago 71s 10.0M 3984M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:10:07.013 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 2m ago 4m 55.2M - 514e6a882f6e ce98632a0fef 2024-04-27T02:10:07.364 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:07 smithi008 ceph-mon[21873]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:07.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:07 smithi077 ceph-mon[24318]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:07.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-27T02:10:08.335 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:08 smithi008 ceph-mon[21873]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:08.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:08 smithi077 ceph-mon[24318]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:09.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:09 smithi008 ceph-mon[21873]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:09.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:09 smithi077 ceph-mon[24318]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:09.641 INFO:teuthology.orchestra.run.smithi008.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-27T02:10:09.641 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:crash 2/2 2m ago 6m * 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:mgr 2/2 2m ago 6m count:2 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:mon 2/2 2m ago 5m smithi008:172.21.15.8=smithi008;smithi077:172.21.15.77=smithi077;count:2 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:osd 8 2m ago - 2024-04-27T02:10:09.642 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-27T02:10:10.267 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-27T02:10:11.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:11 smithi008 ceph-mon[21873]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:11.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:11 smithi008 ceph-mon[21873]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:11 smithi077 ceph-mon[24318]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:11 smithi077 ceph-mon[24318]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:12.232 INFO:teuthology.orchestra.run.smithi008.stdout:HOST ADDR LABELS STATUS 2024-04-27T02:10:12.233 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 172.21.15.8 2024-04-27T02:10:12.233 INFO:teuthology.orchestra.run.smithi008.stdout:smithi077 172.21.15.77 2024-04-27T02:10:12.233 INFO:teuthology.orchestra.run.smithi008.stdout:2 hosts in cluster 2024-04-27T02:10:12.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-27T02:10:13.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:13 smithi008 ceph-mon[21873]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:13.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:13 smithi008 ceph-mon[21873]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:13 smithi077 ceph-mon[24318]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:13 smithi077 ceph-mon[24318]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:14.784 INFO:teuthology.orchestra.run.smithi008.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-27T02:10:14.789 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476004A480DGN 480G LVM detected, locked 2024-04-27T02:10:14.789 INFO:teuthology.orchestra.run.smithi008.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 400G LVM detected, locked 2024-04-27T02:10:15.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:15 smithi008 ceph-mon[21873]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:15.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:15 smithi077 ceph-mon[24318]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:15.488 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T02:10:15.499 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:15.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-27T02:10:15.527 INFO:teuthology.orchestra.run.smithi008.stderr:+ systemctl stop nfs-server 2024-04-27T02:10:15.535 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T02:10:15.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-27T02:10:15.563 INFO:teuthology.orchestra.run.smithi077.stderr:+ systemctl stop nfs-server 2024-04-27T02:10:15.572 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:10:15.581 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:15.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-27T02:10:16.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:16 smithi008 ceph-mon[21873]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:16 smithi077 ceph-mon[24318]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:17.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:17 smithi008 ceph-mon[21873]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:17.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:17 smithi077 ceph-mon[24318]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:18.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:18 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-27T02:10:18.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:18 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-27T02:10:19.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-mon[21873]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:19.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-mon[21873]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:19.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-27T02:10:19.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-mon[21873]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T02:10:19.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-27T02:10:19.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:19 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[21869]: 2024-04-27T02:10:19.132+0000 7f6dd8308700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:10:19.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:19 smithi077 ceph-mon[24318]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:19.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:19 smithi077 ceph-mon[24318]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:19.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:19 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-27T02:10:19.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:19 smithi077 ceph-mon[24318]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T02:10:19.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:19 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-27T02:10:19.814 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-27T02:10:19.825 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-27T02:10:19.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T02:10:20.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: fsmap foofs:0 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: Saving service mds.foofs spec with placement count:2 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:20.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:20.411 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:20 smithi008 ceph-mon[21873]: Deploying daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-27T02:10:20.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: fsmap foofs:0 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: Saving service mds.foofs spec with placement count:2 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:20.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:20 smithi077 ceph-mon[24318]: Deploying daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:10:21.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:21 smithi008 ceph-mon[21873]: pgmap v213: 65 pgs: 10 creating+peering, 6 active+clean, 49 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:21.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:21 smithi008 ceph-mon[21873]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T02:10:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:21 smithi077 ceph-mon[24318]: pgmap v213: 65 pgs: 10 creating+peering, 6 active+clean, 49 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:21 smithi077 ceph-mon[24318]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T02:10:22.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:22 smithi008 ceph-mon[21873]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T02:10:22.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:22 smithi077 ceph-mon[24318]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T02:10:22.977 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:10:22.977 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T02:03:49.512940Z", "last_refresh": "2024-04-27T02:07:28.861856Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T02:03:43.278476Z", "last_refresh": "2024-04-27T02:07:28.861945Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T02:03:46.272790Z", "last_refresh": "2024-04-27T02:07:28.862012Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T02:10:19.154641Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T02:10:19.147042Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T02:03:41.208567Z", "last_refresh": "2024-04-27T02:07:28.861784Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi077:172.21.15.77=smithi077"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T02:04:30.227378Z", "last_refresh": "2024-04-27T02:07:28.861644Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T02:03:48.259486Z", "last_refresh": "2024-04-27T02:07:28.862077Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T02:07:28.862205Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T02:03:44.990122Z", "last_refresh": "2024-04-27T02:07:28.862141Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T02:10:23.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:23 smithi008 ceph-mon[21873]: pgmap v216: 65 pgs: 4 creating+activating, 10 creating+peering, 15 active+clean, 36 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:23.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:23 smithi077 ceph-mon[24318]: pgmap v216: 65 pgs: 4 creating+activating, 10 creating+peering, 15 active+clean, 36 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:23.766 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-27T02:10:24.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:24 smithi008 ceph-mon[21873]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:24.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:24 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:24.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:24 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:10:24.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:24 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:10:24.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:24 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:24.416 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:24 smithi077 ceph-mon[24318]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:24.416 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:24 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:24.416 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:24 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:10:24.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:24 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T02:10:24.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:24 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:24.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: pgmap v217: 65 pgs: 4 creating+activating, 10 creating+peering, 40 active+clean, 11 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: Deploying daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: mds.? [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] up:boot 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: daemon mds.foofs.smithi008.wzyiku assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: Cluster is now healthy 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: fsmap foofs:0 1 up:standby 2024-04-27T02:10:25.386 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:10:25.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:creating} 2024-04-27T02:10:25.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:25 smithi077 ceph-mon[24318]: daemon mds.foofs.smithi008.wzyiku is now active in filesystem foofs as rank 0 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: pgmap v217: 65 pgs: 4 creating+activating, 10 creating+peering, 40 active+clean, 11 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: Deploying daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: mds.? [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] up:boot 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: daemon mds.foofs.smithi008.wzyiku assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: Cluster is now healthy 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: fsmap foofs:0 1 up:standby 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:creating} 2024-04-27T02:10:25.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:25 smithi008 ceph-mon[21873]: daemon mds.foofs.smithi008.wzyiku is now active in filesystem foofs as rank 0 2024-04-27T02:10:26.389 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:26 smithi077 ceph-mon[24318]: mds.? [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] up:active 2024-04-27T02:10:26.390 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:26 smithi077 ceph-mon[24318]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 2024-04-27T02:10:26.392 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:26 smithi008 ceph-mon[21873]: mds.? [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] up:active 2024-04-27T02:10:26.392 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:26 smithi008 ceph-mon[21873]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 2024-04-27T02:10:26.829 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:10:26.829 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T02:03:49.512940Z", "last_refresh": "2024-04-27T02:07:28.861856Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T02:03:43.278476Z", "last_refresh": "2024-04-27T02:07:28.861945Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T02:03:46.272790Z", "last_refresh": "2024-04-27T02:07:28.862012Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T02:10:19.154641Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T02:10:19.147042Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T02:03:41.208567Z", "last_refresh": "2024-04-27T02:07:28.861784Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi077:172.21.15.77=smithi077"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T02:04:30.227378Z", "last_refresh": "2024-04-27T02:07:28.861644Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T02:03:48.259486Z", "last_refresh": "2024-04-27T02:07:28.862077Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T02:07:28.862205Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T02:03:44.990122Z", "last_refresh": "2024-04-27T02:07:28.862141Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T02:10:27.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-27T02:10:27.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:27.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:boot 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:10:27.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:27 smithi008 ceph-mon[21873]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:boot 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:10:27.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:10:27.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:27 smithi077 ceph-mon[24318]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:10:27.828 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-27T02:10:28.642 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:28 smithi008 ceph-mon[21873]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:28.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:28 smithi077 ceph-mon[24318]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:28.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T02:10:29.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:29 smithi008 ceph-mon[21873]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-27T02:10:29.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:29 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:29.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:29 smithi077 ceph-mon[24318]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-27T02:10:29.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:29 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:31.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:31 smithi008 ceph-mon[21873]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T02:10:31.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:31 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:31.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:31 smithi077 ceph-mon[24318]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T02:10:31.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:31 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:32.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:32 smithi008 ceph-mon[21873]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-27T02:10:32.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:32 smithi077 ceph-mon[24318]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-27T02:10:32.884 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:10:32.885 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T02:03:49.512940Z", "last_refresh": "2024-04-27T02:10:32.874113Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T02:03:43.278476Z", "last_refresh": "2024-04-27T02:10:28.414232Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T02:03:46.272790Z", "last_refresh": "2024-04-27T02:10:32.874246Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T02:10:19.154641Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T02:10:19.147042Z", "last_refresh": "2024-04-27T02:10:28.414840Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T02:03:41.208567Z", "last_refresh": "2024-04-27T02:10:28.414378Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi077:172.21.15.77=smithi077"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T02:04:30.227378Z", "last_refresh": "2024-04-27T02:10:28.414453Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T02:03:48.259486Z", "last_refresh": "2024-04-27T02:10:28.414520Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T02:10:28.414585Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T02:03:44.990122Z", "last_refresh": "2024-04-27T02:10:32.874376Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T02:10:33.548 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-27T02:10:33.549 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:10:33.558 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:33.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:33 smithi008 ceph-mon[21873]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:34.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:33 smithi077 ceph-mon[24318]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:10:35.136 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:34 smithi008 ceph-mon[21873]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T02:10:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:34 smithi077 ceph-mon[24318]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T02:10:36.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:36 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-27T02:10:36.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:36 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-27T02:10:37.111 INFO:teuthology.orchestra.run.smithi008.stdout:NFS Cluster Created Successfully 2024-04-27T02:10:37.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T02:10:37.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:37.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-27T02:10:37.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T02:10:37.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-27T02:10:37.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-27T02:10:37.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:37 smithi008 ceph-mon[21873]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-27T02:10:37.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:37 smithi077 ceph-mon[24318]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T02:10:37.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-27T02:10:38.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: Saving service nfs.foo spec with placement count:2 2024-04-27T02:10:38.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: Creating key for client.nfs.foo.0.0.smithi008.lcixut 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:10:38.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T02:10:38.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:38 smithi008 ceph-mon[21873]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: Saving service nfs.foo spec with placement count:2 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: Creating key for client.nfs.foo.0.0.smithi008.lcixut 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:38.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:38 smithi077 ceph-mon[24318]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T02:10:39.336 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:39 smithi008 ceph-mon[21873]: Rados config object exists: conf-nfs.foo 2024-04-27T02:10:39.336 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:39 smithi008 ceph-mon[21873]: Creating key for client.nfs.foo.0.0.smithi008.lcixut-rgw 2024-04-27T02:10:39.336 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:39 smithi008 ceph-mon[21873]: Deploying daemon nfs.foo.0.0.smithi008.lcixut on smithi008 2024-04-27T02:10:39.336 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:39 smithi008 ceph-mon[21873]: pgmap v226: 97 pgs: 5 creating+peering, 27 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-27T02:10:39.336 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:39 smithi008 ceph-mon[21873]: mgrmap e20: smithi008.exkhsc(active, since 5m), standbys: smithi077.snksrp 2024-04-27T02:10:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:39 smithi077 ceph-mon[24318]: Rados config object exists: conf-nfs.foo 2024-04-27T02:10:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:39 smithi077 ceph-mon[24318]: Creating key for client.nfs.foo.0.0.smithi008.lcixut-rgw 2024-04-27T02:10:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:39 smithi077 ceph-mon[24318]: Deploying daemon nfs.foo.0.0.smithi008.lcixut on smithi008 2024-04-27T02:10:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:39 smithi077 ceph-mon[24318]: pgmap v226: 97 pgs: 5 creating+peering, 27 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-27T02:10:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:39 smithi077 ceph-mon[24318]: mgrmap e20: smithi008.exkhsc(active, since 5m), standbys: smithi077.snksrp 2024-04-27T02:10:40.892 INFO:teuthology.orchestra.run.smithi008.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-27T02:10:40.892 INFO:teuthology.orchestra.run.smithi008.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-27T02:10:40.892 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout: "bind": "/fake", 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout: "fs": "foofs", 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster": "foo", 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout: "mode": "RW" 2024-04-27T02:10:41.255 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:10:41.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:41 smithi008 ceph-mon[21873]: pgmap v228: 97 pgs: 5 creating+peering, 14 unknown, 78 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T02:10:41.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:41 smithi077 ceph-mon[24318]: pgmap v228: 97 pgs: 5 creating+peering, 14 unknown, 78 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T02:10:41.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-27T02:10:42.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-27T02:10:42.269 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-27T02:10:42.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:42.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww 2024-04-27T02:10:42.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:10:42.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-27T02:10:42.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-27T02:10:43.518 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:43 smithi008 ceph-mon[21873]: pgmap v229: 97 pgs: 5 creating+peering, 92 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T02:10:43.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:43 smithi077 ceph-mon[24318]: pgmap v229: 97 pgs: 5 creating+peering, 92 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T02:10:44.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:10:44.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:10:44.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:10:44.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T02:10:44.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:44 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:10:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:10:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:10:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T02:10:44.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:44 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:45.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:45 smithi077 ceph-mon[24318]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 136 B/s rd, 272 B/s wr, 0 op/s 2024-04-27T02:10:45.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:45 smithi077 ceph-mon[24318]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:45.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:45 smithi077 ceph-mon[24318]: Rados config object exists: conf-nfs.foo 2024-04-27T02:10:45.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:45 smithi077 ceph-mon[24318]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww-rgw 2024-04-27T02:10:45.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:45 smithi077 ceph-mon[24318]: Deploying daemon nfs.foo.1.0.smithi077.lnlzww on smithi077 2024-04-27T02:10:45.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:45 smithi008 ceph-mon[21873]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 136 B/s rd, 272 B/s wr, 0 op/s 2024-04-27T02:10:45.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:45 smithi008 ceph-mon[21873]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:45.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:45 smithi008 ceph-mon[21873]: Rados config object exists: conf-nfs.foo 2024-04-27T02:10:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:45 smithi008 ceph-mon[21873]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww-rgw 2024-04-27T02:10:45.660 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:45 smithi008 ceph-mon[21873]: Deploying daemon nfs.foo.1.0.smithi077.lnlzww on smithi077 2024-04-27T02:10:47.409 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-27T02:10:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:47.410 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:47 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:47.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:47 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:48.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:48 smithi008 ceph-mon[21873]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:48 smithi077 ceph-mon[24318]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:49 smithi077 ceph-mon[24318]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-27T02:10:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:49 smithi077 ceph-mon[24318]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:49.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:49 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:49.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:49 smithi008 ceph-mon[21873]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-27T02:10:49.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:49 smithi008 ceph-mon[21873]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:49.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:49 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:51.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:50 smithi008 ceph-mon[21873]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:51.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:50 smithi008 ceph-mon[21873]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-27T02:10:51.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:50 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:51.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:50 smithi077 ceph-mon[24318]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:51.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:50 smithi077 ceph-mon[24318]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-27T02:10:51.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:50 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.023 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-27T02:10:52.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:10:52.741 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T02:10:52.752 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:52.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-27T02:10:52.779 INFO:teuthology.orchestra.run.smithi008.stderr:+ mkdir /mnt/foo 2024-04-27T02:10:52.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-27T02:10:52.865 INFO:teuthology.orchestra.run.smithi008.stderr:++ hostname 2024-04-27T02:10:52.866 INFO:teuthology.orchestra.run.smithi008.stderr:+ mount -t nfs smithi008:/fake /mnt/foo -o sync 2024-04-27T02:10:52.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:52 smithi008 ceph-mon[21873]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T02:10:52.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:52 smithi008 ceph-mon[21873]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:53.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:52 smithi077 ceph-mon[24318]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T02:10:53.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:52 smithi077 ceph-mon[24318]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:10:53.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-27T02:10:53.321 INFO:teuthology.orchestra.run.smithi008.stderr:+ echo test 2024-04-27T02:10:53.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-27T02:10:53.426 INFO:teuthology.orchestra.run.smithi008.stderr:+ sync 2024-04-27T02:10:53.654 INFO:teuthology.run_tasks:Running task parallel... 2024-04-27T02:10:53.669 INFO:teuthology.task.parallel:starting parallel... 2024-04-27T02:10:53.670 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T02:10:53.674 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T02:10:53.674 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:53.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-27T02:10:53.675 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T02:10:53.675 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-27T02:10:53.676 INFO:teuthology.task.exec:Executing custom commands... 2024-04-27T02:10:53.676 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:10:53.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-27T02:10:53.733 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-27T02:10:53.733 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:10:53.733 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-27T02:10:53.733 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-27T02:10:53.834 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-27T02:10:53.834 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-04-27T02:10:54.834 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130 28.88 MB/sec warmup 1 sec latency 28.322 ms 2024-04-27T02:10:55.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:54 smithi008 ceph-mon[21873]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T02:10:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:54 smithi077 ceph-mon[24318]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T02:10:55.834 INFO:teuthology.orchestra.run.smithi008.stdout: 5 282 26.27 MB/sec warmup 2 sec latency 25.516 ms 2024-04-27T02:10:56.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-04-27T02:10:56.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-27T02:10:56.070 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-27T02:10:56.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-27T02:10:56.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:10:56.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:55 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:10:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:55 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:10:56.834 INFO:teuthology.orchestra.run.smithi008.stdout: 5 424 24.91 MB/sec warmup 3 sec latency 30.191 ms 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: pgmap v236: 97 pgs: 97 active+clean; 8.9 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 731 KiB/s wr, 33 op/s 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-27T02:10:57.082 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:56 smithi008 ceph-mon[21873]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T02:10:57.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-27T02:10:57.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: pgmap v236: 97 pgs: 97 active+clean; 8.9 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 731 KiB/s wr, 33 op/s 2024-04-27T02:10:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-04-27T02:10:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-27T02:10:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-27T02:10:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-27T02:10:57.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:56 smithi077 ceph-mon[24318]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T02:10:57.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 562 24.08 MB/sec warmup 4 sec latency 55.458 ms 2024-04-27T02:10:58.150 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:57 smithi008 ceph-mon[21873]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T02:10:58.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:57 smithi077 ceph-mon[24318]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T02:10:58.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 870 21.60 MB/sec warmup 5 sec latency 28.615 ms 2024-04-27T02:10:58.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:10:58 smithi008 ceph-mon[21873]: pgmap v239: 97 pgs: 2 peering, 95 active+clean; 23 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s wr, 74 op/s 2024-04-27T02:10:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:10:58 smithi077 ceph-mon[24318]: pgmap v239: 97 pgs: 2 peering, 95 active+clean; 23 MiB data, 52 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s wr, 74 op/s 2024-04-27T02:10:59.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 1225 19.66 MB/sec warmup 6 sec latency 63.151 ms 2024-04-27T02:11:00.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 2190 19.24 MB/sec warmup 7 sec latency 23.709 ms 2024-04-27T02:11:00.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:00 smithi008 ceph-mon[21873]: pgmap v240: 97 pgs: 2 peering, 95 active+clean; 86 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 11 MiB/s wr, 523 op/s 2024-04-27T02:11:01.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:00 smithi077 ceph-mon[24318]: pgmap v240: 97 pgs: 2 peering, 95 active+clean; 86 MiB data, 150 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 11 MiB/s wr, 523 op/s 2024-04-27T02:11:01.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 2981 20.82 MB/sec warmup 8 sec latency 22.765 ms 2024-04-27T02:11:01.866 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-27T02:11:02.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 3733 21.76 MB/sec warmup 9 sec latency 46.449 ms 2024-04-27T02:11:02.960 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:02 smithi008 ceph-mon[21873]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 106 MiB data, 166 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 595 op/s 2024-04-27T02:11:03.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:02 smithi077 ceph-mon[24318]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 106 MiB data, 166 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 595 op/s 2024-04-27T02:11:03.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 4082 20.04 MB/sec warmup 10 sec latency 58.925 ms 2024-04-27T02:11:04.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 4825 19.43 MB/sec warmup 11 sec latency 62.224 ms 2024-04-27T02:11:04.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:04 smithi008 ceph-mon[21873]: pgmap v242: 97 pgs: 97 active+clean; 116 MiB data, 196 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 15 MiB/s wr, 771 op/s 2024-04-27T02:11:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:04 smithi077 ceph-mon[24318]: pgmap v242: 97 pgs: 97 active+clean; 116 MiB data, 196 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 15 MiB/s wr, 771 op/s 2024-04-27T02:11:05.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 5676 19.14 MB/sec warmup 12 sec latency 18.998 ms 2024-04-27T02:11:06.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 6363 19.57 MB/sec warmup 13 sec latency 22.298 ms 2024-04-27T02:11:06.993 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:06 smithi008 ceph-mon[21873]: pgmap v243: 97 pgs: 97 active+clean; 131 MiB data, 215 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 15 MiB/s wr, 901 op/s 2024-04-27T02:11:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:06 smithi077 ceph-mon[24318]: pgmap v243: 97 pgs: 97 active+clean; 131 MiB data, 215 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 15 MiB/s wr, 901 op/s 2024-04-27T02:11:07.392 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph mgr module enable nfs --force' 2024-04-27T02:11:07.841 INFO:teuthology.orchestra.run.smithi008.stdout: 5 7110 20.44 MB/sec warmup 14 sec latency 27.295 ms 2024-04-27T02:11:08.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 7561 19.71 MB/sec warmup 15 sec latency 54.401 ms 2024-04-27T02:11:09.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:08 smithi008 ceph-mon[21873]: pgmap v244: 97 pgs: 97 active+clean; 131 MiB data, 243 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 13 MiB/s wr, 828 op/s 2024-04-27T02:11:09.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:08 smithi077 ceph-mon[24318]: pgmap v244: 97 pgs: 97 active+clean; 131 MiB data, 243 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 13 MiB/s wr, 828 op/s 2024-04-27T02:11:09.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 8129 18.94 MB/sec warmup 16 sec latency 33.185 ms 2024-04-27T02:11:10.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 8749 18.78 MB/sec warmup 17 sec latency 53.790 ms 2024-04-27T02:11:10.944 INFO:teuthology.orchestra.run.smithi008.stderr:module 'nfs' is already enabled 2024-04-27T02:11:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:10 smithi008 ceph-mon[21873]: pgmap v245: 97 pgs: 97 active+clean; 137 MiB data, 320 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 14 MiB/s wr, 995 op/s 2024-04-27T02:11:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:10 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1572650797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T02:11:11.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:10 smithi008 ceph-mon[21873]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T02:11:11.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:10 smithi077 ceph-mon[24318]: pgmap v245: 97 pgs: 97 active+clean; 137 MiB data, 320 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 14 MiB/s wr, 995 op/s 2024-04-27T02:11:11.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:10 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1572650797' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T02:11:11.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:10 smithi077 ceph-mon[24318]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T02:11:11.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 9424 18.30 MB/sec warmup 18 sec latency 31.922 ms 2024-04-27T02:11:12.073 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-27T02:11:12.168 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:11 smithi008 ceph-mon[21873]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-27T02:11:12.168 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:11 smithi008 ceph-mon[21873]: mgrmap e21: smithi008.exkhsc(active, since 6m), standbys: smithi077.snksrp 2024-04-27T02:11:12.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:11 smithi077 ceph-mon[24318]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-27T02:11:12.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:11 smithi077 ceph-mon[24318]: mgrmap e21: smithi008.exkhsc(active, since 6m), standbys: smithi077.snksrp 2024-04-27T02:11:12.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 10012 18.59 MB/sec warmup 19 sec latency 19.915 ms 2024-04-27T02:11:13.190 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:12 smithi008 ceph-mon[21873]: pgmap v246: 97 pgs: 97 active+clean; 138 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 10 MiB/s wr, 739 op/s 2024-04-27T02:11:13.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:12 smithi077 ceph-mon[24318]: pgmap v246: 97 pgs: 97 active+clean; 138 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 10 MiB/s wr, 739 op/s 2024-04-27T02:11:13.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 10703 19.10 MB/sec warmup 20 sec latency 38.614 ms 2024-04-27T02:11:14.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11175 18.66 MB/sec warmup 21 sec latency 32.576 ms 2024-04-27T02:11:14.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:14 smithi008 ceph-mon[21873]: pgmap v247: 97 pgs: 97 active+clean; 147 MiB data, 368 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 934 op/s 2024-04-27T02:11:15.169 INFO:teuthology.orchestra.run.smithi008.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:11:15.221 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:14 smithi077 ceph-mon[24318]: pgmap v247: 97 pgs: 97 active+clean; 147 MiB data, 368 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 934 op/s 2024-04-27T02:11:15.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 11798 18.34 MB/sec warmup 22 sec latency 18.818 ms 2024-04-27T02:11:16.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:16.216 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.217 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:16 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.350 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T02:11:16.351 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:11:16.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:16 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:16.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 12485 18.20 MB/sec warmup 23 sec latency 23.795 ms 2024-04-27T02:11:17.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:17 smithi008 ceph-mon[21873]: pgmap v248: 97 pgs: 97 active+clean; 152 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 944 op/s 2024-04-27T02:11:17.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:17 smithi008 ceph-mon[21873]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:11:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:17 smithi077 ceph-mon[24318]: pgmap v248: 97 pgs: 97 active+clean; 152 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 944 op/s 2024-04-27T02:11:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:17 smithi077 ceph-mon[24318]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T02:11:17.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 13006 17.72 MB/sec warmup 24 sec latency 23.613 ms 2024-04-27T02:11:18.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 13363 17.67 MB/sec warmup 25 sec latency 72.243 ms 2024-04-27T02:11:19.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:19 smithi008 ceph-mon[21873]: pgmap v249: 97 pgs: 97 active+clean; 154 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 755 op/s 2024-04-27T02:11:19.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:19 smithi077 ceph-mon[24318]: pgmap v249: 97 pgs: 97 active+clean; 154 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 755 op/s 2024-04-27T02:11:19.665 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:11:19.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 13833 17.73 MB/sec warmup 26 sec latency 34.218 ms 2024-04-27T02:11:20.794 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:11:20.794 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 29s ago 6m 22.9M - ba2b418f427c aa5e9072673c 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 29s ago 6m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (6m) 31s ago 6m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 29s ago 6m 54.3M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (57s) 29s ago 56s 16.0M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (54s) 31s ago 54s 11.9M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (8m) 29s ago 8m 460M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (6m) 31s ago 6m 413M - 17.2.0 e1d6a67b021e 1c7523763953 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 29s ago 8m 46.9M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (6m) 31s ago 6m 33.8M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (40s) 29s ago 39s 51.1M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (34s) 31s ago 34s 50.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 29s ago 6m 17.4M - 1dbe0e931976 3563103131a3 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (5m) 31s ago 5m 17.1M - 1dbe0e931976 d9d76f2322b0 2024-04-27T02:11:20.795 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 29s ago 5m 41.9M 3217M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 29s ago 4m 43.6M 3217M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 29s ago 4m 39.3M 3217M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 29s ago 3m 40.8M 3217M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (3m) 31s ago 3m 41.8M 3984M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (3m) 31s ago 3m 38.4M 3984M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (2m) 31s ago 2m 39.0M 3984M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (2m) 31s ago 2m 40.5M 3984M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:11:20.796 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 29s ago 5m 62.5M - 514e6a882f6e ce98632a0fef 2024-04-27T02:11:20.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14269 17.70 MB/sec warmup 27 sec latency 31.434 ms 2024-04-27T02:11:21.259 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:21 smithi008 ceph-mon[21873]: pgmap v250: 97 pgs: 97 active+clean; 155 MiB data, 461 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 934 op/s 2024-04-27T02:11:21.260 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:21 smithi008 ceph-mon[21873]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:21.260 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:21 smithi008 ceph-mon[21873]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:21 smithi077 ceph-mon[24318]: pgmap v250: 97 pgs: 97 active+clean; 155 MiB data, 461 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 934 op/s 2024-04-27T02:11:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:21 smithi077 ceph-mon[24318]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:21 smithi077 ceph-mon[24318]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:21.510 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:11:21.510 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:11:21.510 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:21.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:21.510 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:11:21.511 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:21.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:21.512 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:11:21.512 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T02:11:21.512 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:11:21.512 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:11:21.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14523 17.36 MB/sec warmup 28 sec latency 45.553 ms 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "", 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a image" 2024-04-27T02:11:22.090 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:11:22.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:22 smithi008 ceph-mon[21873]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:22.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:22 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3755143820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:11:22.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:22 smithi077 ceph-mon[24318]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:22.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:22 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3755143820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:11:22.797 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:11:22.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 14788 16.82 MB/sec warmup 29 sec latency 37.044 ms 2024-04-27T02:11:23.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:23 smithi008 ceph-mon[21873]: pgmap v251: 97 pgs: 97 active+clean; 153 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 714 op/s 2024-04-27T02:11:23.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:23 smithi008 ceph-mon[21873]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:23.274 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:23 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/947810086' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:11:23.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:23 smithi077 ceph-mon[24318]: pgmap v251: 97 pgs: 97 active+clean; 153 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 714 op/s 2024-04-27T02:11:23.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:23 smithi077 ceph-mon[24318]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:23.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:23 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/947810086' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:11:23.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 15402 16.67 MB/sec warmup 30 sec latency 22.004 ms 2024-04-27T02:11:24.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 16132 16.63 MB/sec warmup 31 sec latency 25.433 ms 2024-04-27T02:11:25.288 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:25 smithi008 ceph-mon[21873]: pgmap v252: 97 pgs: 97 active+clean; 159 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.9 MiB/s wr, 761 op/s 2024-04-27T02:11:25.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:25 smithi077 ceph-mon[24318]: pgmap v252: 97 pgs: 97 active+clean; 159 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.9 MiB/s wr, 761 op/s 2024-04-27T02:11:25.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 16648 16.44 MB/sec warmup 32 sec latency 40.208 ms 2024-04-27T02:11:26.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 17183 16.62 MB/sec warmup 33 sec latency 23.498 ms 2024-04-27T02:11:27.312 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:27 smithi008 ceph-mon[21873]: pgmap v253: 97 pgs: 97 active+clean; 170 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.9 MiB/s wr, 710 op/s 2024-04-27T02:11:27.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:27 smithi077 ceph-mon[24318]: pgmap v253: 97 pgs: 97 active+clean; 170 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 7.9 MiB/s wr, 710 op/s 2024-04-27T02:11:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 17742 16.77 MB/sec warmup 34 sec latency 32.250 ms 2024-04-27T02:11:28.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18247 16.66 MB/sec warmup 35 sec latency 15.870 ms 2024-04-27T02:11:29.320 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:29 smithi008 ceph-mon[21873]: pgmap v254: 97 pgs: 97 active+clean; 167 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 581 op/s 2024-04-27T02:11:29.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:29 smithi077 ceph-mon[24318]: pgmap v254: 97 pgs: 97 active+clean; 167 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 581 op/s 2024-04-27T02:11:29.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 18928 16.54 MB/sec warmup 36 sec latency 23.399 ms 2024-04-27T02:11:30.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 19754 16.53 MB/sec warmup 37 sec latency 54.833 ms 2024-04-27T02:11:31.334 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:31 smithi008 ceph-mon[21873]: pgmap v255: 97 pgs: 97 active+clean; 171 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 746 op/s 2024-04-27T02:11:31.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:31 smithi077 ceph-mon[24318]: pgmap v255: 97 pgs: 97 active+clean; 171 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 746 op/s 2024-04-27T02:11:31.842 INFO:teuthology.orchestra.run.smithi008.stdout: 5 20640 16.76 MB/sec warmup 38 sec latency 18.701 ms 2024-04-27T02:11:32.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 21501 17.28 MB/sec warmup 39 sec latency 19.294 ms 2024-04-27T02:11:33.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:33 smithi008 ceph-mon[21873]: pgmap v256: 97 pgs: 97 active+clean; 175 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.8 MiB/s wr, 574 op/s 2024-04-27T02:11:33.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:33 smithi077 ceph-mon[24318]: pgmap v256: 97 pgs: 97 active+clean; 175 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.8 MiB/s wr, 574 op/s 2024-04-27T02:11:33.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 21948 16.99 MB/sec warmup 40 sec latency 31.856 ms 2024-04-27T02:11:34.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 22689 16.96 MB/sec warmup 41 sec latency 25.655 ms 2024-04-27T02:11:35.340 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:35 smithi008 ceph-mon[21873]: pgmap v257: 97 pgs: 97 active+clean; 181 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 691 op/s 2024-04-27T02:11:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:35 smithi077 ceph-mon[24318]: pgmap v257: 97 pgs: 97 active+clean; 181 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 691 op/s 2024-04-27T02:11:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 23395 16.85 MB/sec warmup 42 sec latency 26.237 ms 2024-04-27T02:11:36.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 23912 16.77 MB/sec warmup 43 sec latency 30.559 ms 2024-04-27T02:11:37.372 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:37 smithi008 ceph-mon[21873]: pgmap v258: 97 pgs: 97 active+clean; 189 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 858 op/s 2024-04-27T02:11:37.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:37 smithi077 ceph-mon[24318]: pgmap v258: 97 pgs: 97 active+clean; 189 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 858 op/s 2024-04-27T02:11:37.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 24595 17.04 MB/sec warmup 44 sec latency 28.452 ms 2024-04-27T02:11:38.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25134 17.13 MB/sec warmup 45 sec latency 32.639 ms 2024-04-27T02:11:39.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:39 smithi008 ceph-mon[21873]: pgmap v259: 97 pgs: 97 active+clean; 187 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 699 op/s 2024-04-27T02:11:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:39 smithi077 ceph-mon[24318]: pgmap v259: 97 pgs: 97 active+clean; 187 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 699 op/s 2024-04-27T02:11:39.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 25641 16.89 MB/sec warmup 46 sec latency 64.536 ms 2024-04-27T02:11:40.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 26103 16.74 MB/sec warmup 47 sec latency 39.104 ms 2024-04-27T02:11:41.445 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:41 smithi008 ceph-mon[21873]: pgmap v260: 97 pgs: 97 active+clean; 195 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 959 op/s 2024-04-27T02:11:41.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:41 smithi077 ceph-mon[24318]: pgmap v260: 97 pgs: 97 active+clean; 195 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 959 op/s 2024-04-27T02:11:41.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 26736 16.71 MB/sec warmup 48 sec latency 18.682 ms 2024-04-27T02:11:42.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 27491 16.68 MB/sec warmup 49 sec latency 18.937 ms 2024-04-27T02:11:43.458 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:43 smithi008 ceph-mon[21873]: pgmap v261: 97 pgs: 97 active+clean; 198 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 882 op/s 2024-04-27T02:11:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:43 smithi077 ceph-mon[24318]: pgmap v261: 97 pgs: 97 active+clean; 198 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 882 op/s 2024-04-27T02:11:43.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 28248 16.96 MB/sec warmup 50 sec latency 22.488 ms 2024-04-27T02:11:44.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 28793 17.05 MB/sec warmup 51 sec latency 23.118 ms 2024-04-27T02:11:45.476 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:45 smithi008 ceph-mon[21873]: pgmap v262: 97 pgs: 97 active+clean; 201 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 973 op/s 2024-04-27T02:11:45.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:45 smithi077 ceph-mon[24318]: pgmap v262: 97 pgs: 97 active+clean; 201 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 973 op/s 2024-04-27T02:11:45.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 29311 16.80 MB/sec warmup 52 sec latency 44.559 ms 2024-04-27T02:11:46.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 29801 16.78 MB/sec warmup 53 sec latency 37.578 ms 2024-04-27T02:11:47.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:47 smithi008 ceph-mon[21873]: pgmap v263: 97 pgs: 97 active+clean; 205 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 989 op/s 2024-04-27T02:11:47.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:47 smithi077 ceph-mon[24318]: pgmap v263: 97 pgs: 97 active+clean; 205 MiB data, 847 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 989 op/s 2024-04-27T02:11:47.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 30496 16.65 MB/sec warmup 54 sec latency 37.807 ms 2024-04-27T02:11:48.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 31214 16.75 MB/sec warmup 55 sec latency 36.502 ms 2024-04-27T02:11:49.275 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:49 smithi008 ceph-mon[21873]: pgmap v264: 97 pgs: 97 active+clean; 211 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 780 op/s 2024-04-27T02:11:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:49 smithi077 ceph-mon[24318]: pgmap v264: 97 pgs: 97 active+clean; 211 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 780 op/s 2024-04-27T02:11:49.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 31956 17.01 MB/sec warmup 56 sec latency 22.827 ms 2024-04-27T02:11:50.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32447 16.94 MB/sec warmup 57 sec latency 30.945 ms 2024-04-27T02:11:51.462 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:51 smithi008 ceph-mon[21873]: pgmap v265: 97 pgs: 97 active+clean; 220 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-27T02:11:51.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:51.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:11:51.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:51 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:51 smithi077 ceph-mon[24318]: pgmap v265: 97 pgs: 97 active+clean; 220 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-27T02:11:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:11:51.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:51 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:11:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 32958 16.73 MB/sec warmup 58 sec latency 22.937 ms 2024-04-27T02:11:52.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:52 smithi008 ceph-mon[21873]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T02:11:52.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:52 smithi008 ceph-mon[21873]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T02:11:52.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:52 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi008.exkhsc) 2024-04-27T02:11:52.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:52 smithi008 ceph-mon[21873]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi077 2024-04-27T02:11:52.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:52 smithi077 ceph-mon[24318]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T02:11:52.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:52 smithi077 ceph-mon[24318]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T02:11:52.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:52 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi008.exkhsc) 2024-04-27T02:11:52.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:52 smithi077 ceph-mon[24318]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi077 2024-04-27T02:11:52.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 33547 16.80 MB/sec warmup 59 sec latency 43.890 ms 2024-04-27T02:11:53.385 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:11:53.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:53 smithi008 ceph-mon[21873]: pgmap v266: 97 pgs: 97 active+clean; 218 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 790 op/s 2024-04-27T02:11:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:53 smithi077 ceph-mon[24318]: pgmap v266: 97 pgs: 97 active+clean; 218 MiB data, 942 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 790 op/s 2024-04-27T02:11:53.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 34257 16.67 MB/sec warmup 60 sec latency 42.939 ms 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 62s ago 7m 22.9M - ba2b418f427c aa5e9072673c 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 62s ago 7m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (6m) 65s ago 6m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (5m) 62s ago 7m 54.3M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (91s) 62s ago 90s 16.0M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (88s) 65s ago 87s 11.9M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (8m) 62s ago 8m 460M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (6m) 65s ago 6m 413M - 17.2.0 e1d6a67b021e 1c7523763953 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 62s ago 8m 46.9M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (6m) 65s ago 6m 33.8M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:11:54.317 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (73s) 62s ago 73s 51.1M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (68s) 65s ago 67s 50.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 62s ago 7m 17.4M - 1dbe0e931976 3563103131a3 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (6m) 65s ago 6m 17.1M - 1dbe0e931976 d9d76f2322b0 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 62s ago 5m 41.9M 3217M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 62s ago 5m 43.6M 3217M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 62s ago 4m 39.3M 3217M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 62s ago 4m 40.8M 3217M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (4m) 65s ago 4m 41.8M 3984M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (3m) 65s ago 3m 38.4M 3984M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (3m) 65s ago 3m 39.0M 3984M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:11:54.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (2m) 65s ago 2m 40.5M 3984M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:11:54.319 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 62s ago 6m 62.5M - 514e6a882f6e ce98632a0fef 2024-04-27T02:11:54.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 34891 16.77 MB/sec warmup 61 sec latency 30.217 ms 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:11:54.851 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:11:54.852 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 image on host smithi077" 2024-04-27T02:11:55.395 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:11:55.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:55 smithi008 ceph-mon[21873]: from='client.24417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:55.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:55 smithi008 ceph-mon[21873]: pgmap v267: 97 pgs: 97 active+clean; 223 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 775 op/s 2024-04-27T02:11:55.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:55 smithi008 ceph-mon[21873]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:55.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:55 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/750813523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:11:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:55 smithi077 ceph-mon[24318]: from='client.24417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:55 smithi077 ceph-mon[24318]: pgmap v267: 97 pgs: 97 active+clean; 223 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 775 op/s 2024-04-27T02:11:55.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:55 smithi077 ceph-mon[24318]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:55.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:55 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/750813523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:11:55.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 35556 16.92 MB/sec warmup 62 sec latency 59.392 ms 2024-04-27T02:11:55.904 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:11:56.518 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:56 smithi008 ceph-mon[21873]: from='client.24425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:56.519 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:11:56.519 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:56 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:11:56.519 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:56 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2517579150' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:11:56.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:56 smithi077 ceph-mon[24318]: from='client.24425 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:56.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:11:56.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:56 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:11:56.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:56 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2517579150' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:11:56.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 35942 16.84 MB/sec warmup 63 sec latency 69.384 ms 2024-04-27T02:11:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:57 smithi008 ceph-mon[21873]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:57.283 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:57 smithi008 ceph-mon[21873]: pgmap v268: 97 pgs: 97 active+clean; 227 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 833 op/s 2024-04-27T02:11:57.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:57 smithi077 ceph-mon[24318]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:11:57.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:57 smithi077 ceph-mon[24318]: pgmap v268: 97 pgs: 97 active+clean; 227 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 833 op/s 2024-04-27T02:11:57.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 36328 16.63 MB/sec warmup 64 sec latency 36.498 ms 2024-04-27T02:11:58.539 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:11:58 smithi008 ceph-mon[21873]: pgmap v269: 97 pgs: 97 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 693 op/s 2024-04-27T02:11:58.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:11:58 smithi077 ceph-mon[24318]: pgmap v269: 97 pgs: 97 active+clean; 225 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 693 op/s 2024-04-27T02:11:58.843 INFO:teuthology.orchestra.run.smithi008.stdout: 5 36996 16.63 MB/sec warmup 65 sec latency 32.656 ms 2024-04-27T02:11:59.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 37829 16.60 MB/sec warmup 66 sec latency 25.008 ms 2024-04-27T02:12:00.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 38555 16.80 MB/sec warmup 67 sec latency 22.201 ms 2024-04-27T02:12:01.056 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:00 smithi008 ceph-mon[21873]: pgmap v270: 97 pgs: 97 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 907 op/s 2024-04-27T02:12:01.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:00 smithi077 ceph-mon[24318]: pgmap v270: 97 pgs: 97 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 907 op/s 2024-04-27T02:12:01.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39238 16.93 MB/sec warmup 68 sec latency 19.889 ms 2024-04-27T02:12:02.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 39618 16.78 MB/sec warmup 69 sec latency 31.709 ms 2024-04-27T02:12:03.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:02 smithi008 ceph-mon[21873]: pgmap v271: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.8 MiB/s wr, 666 op/s 2024-04-27T02:12:03.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:02 smithi077 ceph-mon[24318]: pgmap v271: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.8 MiB/s wr, 666 op/s 2024-04-27T02:12:03.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 40204 16.67 MB/sec warmup 70 sec latency 22.909 ms 2024-04-27T02:12:04.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 40812 16.68 MB/sec warmup 71 sec latency 49.860 ms 2024-04-27T02:12:05.085 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:04 smithi008 ceph-mon[21873]: pgmap v272: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 895 op/s 2024-04-27T02:12:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:04 smithi077 ceph-mon[24318]: pgmap v272: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 895 op/s 2024-04-27T02:12:05.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 41533 16.62 MB/sec warmup 72 sec latency 43.685 ms 2024-04-27T02:12:06.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 42508 16.90 MB/sec warmup 73 sec latency 23.594 ms 2024-04-27T02:12:07.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:06 smithi008 ceph-mon[21873]: pgmap v273: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 9.2 MiB/s wr, 873 op/s 2024-04-27T02:12:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:06 smithi077 ceph-mon[24318]: pgmap v273: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 9.2 MiB/s wr, 873 op/s 2024-04-27T02:12:07.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43162 16.95 MB/sec warmup 74 sec latency 19.204 ms 2024-04-27T02:12:08.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 43800 16.85 MB/sec warmup 75 sec latency 28.730 ms 2024-04-27T02:12:08.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:08 smithi008 ceph-mon[21873]: pgmap v274: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-04-27T02:12:09.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:08 smithi077 ceph-mon[24318]: pgmap v274: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.9 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-04-27T02:12:09.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 44663 16.91 MB/sec warmup 76 sec latency 28.442 ms 2024-04-27T02:12:10.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 45546 17.04 MB/sec warmup 77 sec latency 17.817 ms 2024-04-27T02:12:11.010 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:10 smithi008 ceph-mon[21873]: pgmap v275: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-04-27T02:12:11.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:10 smithi077 ceph-mon[24318]: pgmap v275: 97 pgs: 97 active+clean; 261 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-04-27T02:12:11.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 46505 17.27 MB/sec warmup 78 sec latency 15.124 ms 2024-04-27T02:12:12.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47167 17.16 MB/sec warmup 79 sec latency 17.421 ms 2024-04-27T02:12:13.021 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:12 smithi008 ceph-mon[21873]: pgmap v276: 97 pgs: 97 active+clean; 266 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 875 op/s 2024-04-27T02:12:13.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:12 smithi077 ceph-mon[24318]: pgmap v276: 97 pgs: 97 active+clean; 266 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 875 op/s 2024-04-27T02:12:13.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 47924 17.20 MB/sec warmup 80 sec latency 23.331 ms 2024-04-27T02:12:14.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 48818 17.22 MB/sec warmup 81 sec latency 23.243 ms 2024-04-27T02:12:14.992 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:14 smithi008 ceph-mon[21873]: pgmap v277: 97 pgs: 97 active+clean; 266 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 923 op/s 2024-04-27T02:12:15.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:14 smithi077 ceph-mon[24318]: pgmap v277: 97 pgs: 97 active+clean; 266 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.9 MiB/s wr, 923 op/s 2024-04-27T02:12:15.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 49393 17.30 MB/sec warmup 82 sec latency 56.333 ms 2024-04-27T02:12:16.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50033 17.37 MB/sec warmup 83 sec latency 23.181 ms 2024-04-27T02:12:17.074 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:16 smithi008 ceph-mon[21873]: pgmap v278: 97 pgs: 97 active+clean; 272 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 864 op/s 2024-04-27T02:12:17.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:16 smithi077 ceph-mon[24318]: pgmap v278: 97 pgs: 97 active+clean; 272 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 864 op/s 2024-04-27T02:12:17.844 INFO:teuthology.orchestra.run.smithi008.stdout: 5 50573 17.26 MB/sec warmup 84 sec latency 22.824 ms 2024-04-27T02:12:18.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 51191 17.20 MB/sec warmup 85 sec latency 47.063 ms 2024-04-27T02:12:19.090 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:18 smithi008 ceph-mon[21873]: pgmap v279: 97 pgs: 97 active+clean; 276 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 899 op/s 2024-04-27T02:12:19.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:18 smithi077 ceph-mon[24318]: pgmap v279: 97 pgs: 97 active+clean; 276 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 899 op/s 2024-04-27T02:12:19.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 52036 17.21 MB/sec warmup 86 sec latency 22.055 ms 2024-04-27T02:12:20.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 52749 17.34 MB/sec warmup 87 sec latency 42.194 ms 2024-04-27T02:12:21.109 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:20 smithi008 ceph-mon[21873]: pgmap v280: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-27T02:12:21.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:20 smithi077 ceph-mon[24318]: pgmap v280: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-27T02:12:21.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53492 17.44 MB/sec warmup 88 sec latency 23.327 ms 2024-04-27T02:12:22.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 53969 17.35 MB/sec warmup 89 sec latency 26.706 ms 2024-04-27T02:12:23.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:22 smithi008 ceph-mon[21873]: pgmap v281: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 865 op/s 2024-04-27T02:12:23.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:22 smithi077 ceph-mon[24318]: pgmap v281: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 865 op/s 2024-04-27T02:12:23.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 54493 17.22 MB/sec warmup 90 sec latency 88.126 ms 2024-04-27T02:12:24.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 55064 17.23 MB/sec warmup 91 sec latency 51.432 ms 2024-04-27T02:12:25.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:24 smithi008 ceph-mon[21873]: pgmap v282: 97 pgs: 97 active+clean; 291 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-04-27T02:12:25.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:24 smithi077 ceph-mon[24318]: pgmap v282: 97 pgs: 97 active+clean; 291 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-04-27T02:12:25.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 55818 17.20 MB/sec warmup 92 sec latency 25.067 ms 2024-04-27T02:12:26.340 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:12:26.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 56397 17.28 MB/sec warmup 93 sec latency 76.510 ms 2024-04-27T02:12:27.111 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:26 smithi008 ceph-mon[21873]: pgmap v283: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 996 op/s 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 95s ago 8m 22.9M - ba2b418f427c aa5e9072673c 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 95s ago 7m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (7m) 97s ago 7m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 95s ago 7m 54.3M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (2m) 95s ago 2m 16.0M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (2m) 97s ago 2m 11.9M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (9m) 95s ago 9m 460M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (7m) 97s ago 7m 413M - 17.2.0 e1d6a67b021e 1c7523763953 2024-04-27T02:12:27.192 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 95s ago 9m 46.9M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (7m) 97s ago 7m 33.8M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (106s) 95s ago 106s 51.1M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (101s) 97s ago 100s 50.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 95s ago 7m 17.4M - 1dbe0e931976 3563103131a3 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (7m) 97s ago 7m 17.1M - 1dbe0e931976 d9d76f2322b0 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 95s ago 6m 41.9M 3217M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 95s ago 5m 43.6M 3217M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 95s ago 5m 39.3M 3217M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 95s ago 5m 40.8M 3217M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (4m) 97s ago 4m 41.8M 3984M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (4m) 97s ago 4m 38.4M 3984M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (3m) 97s ago 3m 39.0M 3984M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (3m) 97s ago 3m 40.5M 3984M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:12:27.193 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 95s ago 6m 62.5M - 514e6a882f6e ce98632a0fef 2024-04-27T02:12:27.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:26 smithi077 ceph-mon[24318]: pgmap v283: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 996 op/s 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:12:27.735 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:12:27.736 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:12:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57002 17.34 MB/sec warmup 94 sec latency 32.911 ms 2024-04-27T02:12:28.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:28.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:12:28.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:12:28.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:12:28.120 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:27 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/37885904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:12:28.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:12:28.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:27 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/37885904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:12:28.273 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons" 2024-04-27T02:12:28.274 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:12:28.801 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:12:28.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57433 17.23 MB/sec warmup 95 sec latency 30.988 ms 2024-04-27T02:12:29.133 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:28 smithi008 ceph-mon[21873]: pgmap v284: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 854 op/s 2024-04-27T02:12:29.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:28 smithi008 ceph-mon[21873]: Upgrade: Updating mgr.smithi077.snksrp 2024-04-27T02:12:29.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:28 smithi008 ceph-mon[21873]: Deploying daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:12:29.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:28 smithi008 ceph-mon[21873]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:29.134 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:28 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1825244742' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:12:29.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:28 smithi077 ceph-mon[24318]: pgmap v284: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 854 op/s 2024-04-27T02:12:29.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:28 smithi077 ceph-mon[24318]: Upgrade: Updating mgr.smithi077.snksrp 2024-04-27T02:12:29.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:28 smithi077 ceph-mon[24318]: Deploying daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:12:29.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:28 smithi077 ceph-mon[24318]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:12:29.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:28 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1825244742' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:12:29.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 57931 17.13 MB/sec warmup 96 sec latency 49.439 ms 2024-04-27T02:12:30.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 58389 17.08 MB/sec warmup 97 sec latency 33.820 ms 2024-04-27T02:12:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:30 smithi077 ceph-mon[24318]: pgmap v285: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-27T02:12:31.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:30 smithi008 ceph-mon[21873]: pgmap v285: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-27T02:12:31.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 59055 17.05 MB/sec warmup 98 sec latency 36.932 ms 2024-04-27T02:12:32.414 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:32.414 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:32.414 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:12:32.415 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:32 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:12:32.415 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:32 smithi008 ceph-mon[21873]: pgmap v286: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 783 op/s 2024-04-27T02:12:32.457 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:32.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:32.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:12:32.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:32 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:12:32.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:32 smithi077 ceph-mon[24318]: pgmap v286: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 783 op/s 2024-04-27T02:12:32.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 59605 17.05 MB/sec warmup 99 sec latency 25.655 ms 2024-04-27T02:12:33.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 60245 17.12 MB/sec warmup 100 sec latency 29.098 ms 2024-04-27T02:12:34.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:34 smithi008 ceph-mon[21873]: pgmap v287: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 886 op/s 2024-04-27T02:12:34.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 60712 17.12 MB/sec warmup 101 sec latency 37.006 ms 2024-04-27T02:12:34.862 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:34 smithi077 ceph-mon[24318]: pgmap v287: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 886 op/s 2024-04-27T02:12:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61096 17.02 MB/sec warmup 102 sec latency 28.186 ms 2024-04-27T02:12:36.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:36 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:36.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:36 smithi008 ceph-mon[21873]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 741 op/s 2024-04-27T02:12:36.450 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:36 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:36.582 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:36 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:36.583 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:36 smithi077 ceph-mon[24318]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 741 op/s 2024-04-27T02:12:36.583 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:36 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:36.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61545 16.90 MB/sec warmup 103 sec latency 41.547 ms 2024-04-27T02:12:37.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 61971 16.85 MB/sec warmup 104 sec latency 106.519 ms 2024-04-27T02:12:38.734 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:38 smithi008 ceph-mon[21873]: pgmap v289: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 729 op/s 2024-04-27T02:12:38.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 62661 16.84 MB/sec warmup 105 sec latency 43.843 ms 2024-04-27T02:12:38.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:38 smithi077 ceph-mon[24318]: pgmap v289: 97 pgs: 97 active+clean; 314 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 729 op/s 2024-04-27T02:12:39.743 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:12:39.743 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:12:39.744 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:12:39.744 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:12:39.744 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:12:39.744 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:39 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:12:39.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63173 16.83 MB/sec warmup 106 sec latency 38.900 ms 2024-04-27T02:12:39.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:12:39.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:12:39.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:12:39.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:12:39.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:12:39.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:39 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:12:40.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:40 smithi008 ceph-mon[21873]: pgmap v290: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 903 op/s 2024-04-27T02:12:40.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:40 smithi008 ceph-mon[21873]: mgrmap e22: smithi008.exkhsc(active, since 7m), standbys: smithi077.snksrp 2024-04-27T02:12:40.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 63852 16.90 MB/sec warmup 107 sec latency 25.078 ms 2024-04-27T02:12:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:40 smithi077 ceph-mon[24318]: pgmap v290: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 903 op/s 2024-04-27T02:12:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:40 smithi077 ceph-mon[24318]: mgrmap e22: smithi008.exkhsc(active, since 7m), standbys: smithi077.snksrp 2024-04-27T02:12:41.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64336 16.91 MB/sec warmup 108 sec latency 25.626 ms 2024-04-27T02:12:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi008.exkhsc) 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: Failing over to other MGR 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr fail", "who": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T02:12:42.034 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:42 smithi008 ceph-mon[21873]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi008.exkhsc) 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: Failing over to other MGR 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr fail", "who": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:12:42.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T02:12:42.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:42 smithi077 ceph-mon[24318]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:12:42.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 64739 16.81 MB/sec warmup 109 sec latency 36.369 ms 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "mgr fail", "who": "smithi008.exkhsc"}]': finished 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: mgrmap e23: smithi077.snksrp(active, starting, since 0.99849s), standbys: smithi008.exkhsc 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:12:43.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:12:43.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: Manager daemon smithi077.snksrp is now available 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/trash_purge_schedule"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/trash_purge_schedule"}]: dispatch 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.298 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:43 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14206 172.21.15.8:0/3804466069' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "mgr fail", "who": "smithi008.exkhsc"}]': finished 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: mgrmap e23: smithi077.snksrp(active, starting, since 0.99849s), standbys: smithi008.exkhsc 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:12:43.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:12:43.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: Manager daemon smithi077.snksrp is now available 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/trash_purge_schedule"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.snksrp/trash_purge_schedule"}]: dispatch 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:43 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:43.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 65155 16.71 MB/sec warmup 110 sec latency 37.534 ms 2024-04-27T02:12:44.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: Deploying cephadm binary to smithi077 2024-04-27T02:12:44.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: Deploying cephadm binary to smithi008 2024-04-27T02:12:44.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: [27/Apr/2024:02:12:42] ENGINE Bus STARTING 2024-04-27T02:12:44.300 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: [27/Apr/2024:02:12:42] ENGINE Serving on https://172.21.15.77:7150 2024-04-27T02:12:44.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: [27/Apr/2024:02:12:42] ENGINE Serving on http://172.21.15.77:8765 2024-04-27T02:12:44.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: [27/Apr/2024:02:12:42] ENGINE Bus STARTED 2024-04-27T02:12:44.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: mgrmap e24: smithi077.snksrp(active, since 2s), standbys: smithi008.exkhsc 2024-04-27T02:12:44.301 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:44 smithi008 ceph-mon[21873]: pgmap v3: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:44.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: Deploying cephadm binary to smithi077 2024-04-27T02:12:44.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: Deploying cephadm binary to smithi008 2024-04-27T02:12:44.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: [27/Apr/2024:02:12:42] ENGINE Bus STARTING 2024-04-27T02:12:44.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: [27/Apr/2024:02:12:42] ENGINE Serving on https://172.21.15.77:7150 2024-04-27T02:12:44.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: [27/Apr/2024:02:12:42] ENGINE Serving on http://172.21.15.77:8765 2024-04-27T02:12:44.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: [27/Apr/2024:02:12:42] ENGINE Bus STARTED 2024-04-27T02:12:44.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: mgrmap e24: smithi077.snksrp(active, since 2s), standbys: smithi008.exkhsc 2024-04-27T02:12:44.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:44 smithi077 ceph-mon[24318]: pgmap v3: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:44.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 65479 16.65 MB/sec warmup 111 sec latency 167.504 ms 2024-04-27T02:12:45.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:45 smithi008 ceph-mon[21873]: pgmap v4: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:45.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:45 smithi008 ceph-mon[21873]: mgrmap e25: smithi077.snksrp(active, since 3s), standbys: smithi008.exkhsc 2024-04-27T02:12:45.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:45 smithi077 ceph-mon[24318]: pgmap v4: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:45.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:45 smithi077 ceph-mon[24318]: mgrmap e25: smithi077.snksrp(active, since 3s), standbys: smithi008.exkhsc 2024-04-27T02:12:45.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66032 16.62 MB/sec warmup 112 sec latency 55.404 ms 2024-04-27T02:12:46.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66550 16.59 MB/sec warmup 113 sec latency 51.031 ms 2024-04-27T02:12:47.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:47 smithi008 ceph-mon[21873]: pgmap v5: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:47.241 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:47 smithi008 ceph-mon[21873]: mgrmap e26: smithi077.snksrp(active, since 5s), standbys: smithi008.exkhsc 2024-04-27T02:12:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:47 smithi077 ceph-mon[24318]: pgmap v5: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:47 smithi077 ceph-mon[24318]: mgrmap e26: smithi077.snksrp(active, since 5s), standbys: smithi008.exkhsc 2024-04-27T02:12:47.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 66990 16.59 MB/sec warmup 114 sec latency 24.090 ms 2024-04-27T02:12:48.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67548 16.61 MB/sec warmup 115 sec latency 51.217 ms 2024-04-27T02:12:49.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:49 smithi077 ceph-mon[24318]: pgmap v6: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:49.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:49 smithi008 ceph-mon[21873]: pgmap v6: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:12:49.845 INFO:teuthology.orchestra.run.smithi008.stdout: 5 67882 16.57 MB/sec warmup 116 sec latency 82.722 ms 2024-04-27T02:12:50.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68220 16.47 MB/sec warmup 117 sec latency 70.968 ms 2024-04-27T02:12:51.068 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:50 smithi008 ceph-mon[21873]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.8 MiB/s wr, 606 op/s 2024-04-27T02:12:51.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:50 smithi077 ceph-mon[24318]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 4.8 MiB/s wr, 606 op/s 2024-04-27T02:12:51.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 68612 16.37 MB/sec warmup 118 sec latency 54.066 ms 2024-04-27T02:12:52.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: Standby manager daemon smithi008.exkhsc restarted 2024-04-27T02:12:52.072 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:12:52.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/crt"}]: dispatch 2024-04-27T02:12:52.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:12:52.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/key"}]: dispatch 2024-04-27T02:12:52.073 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:51 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:12:52.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: Standby manager daemon smithi008.exkhsc restarted 2024-04-27T02:12:52.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:12:52.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/crt"}]: dispatch 2024-04-27T02:12:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:12:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/key"}]: dispatch 2024-04-27T02:12:52.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:51 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/2176067409' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:12:52.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 69115 16.37 MB/sec warmup 119 sec latency 34.648 ms 2024-04-27T02:12:53.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:52 smithi008 ceph-mon[21873]: mgrmap e27: smithi077.snksrp(active, since 10s), standbys: smithi008.exkhsc 2024-04-27T02:12:53.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:52 smithi008 ceph-mon[21873]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.7 MiB/s wr, 470 op/s 2024-04-27T02:12:53.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:52 smithi077 ceph-mon[24318]: mgrmap e27: smithi077.snksrp(active, since 10s), standbys: smithi008.exkhsc 2024-04-27T02:12:53.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:52 smithi077 ceph-mon[24318]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.7 MiB/s wr, 470 op/s 2024-04-27T02:12:54.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 70292 13.15 MB/sec execute 1 sec latency 64.266 ms 2024-04-27T02:12:55.354 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:55 smithi008 ceph-mon[21873]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.0 MiB/s wr, 684 op/s 2024-04-27T02:12:55.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:55 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:55.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:55 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:55.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:55 smithi077 ceph-mon[24318]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 6.0 MiB/s wr, 684 op/s 2024-04-27T02:12:55.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:55 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:55.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:55 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:55.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 70948 18.65 MB/sec execute 2 sec latency 22.764 ms 2024-04-27T02:12:56.617 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:56 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.618 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:56 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.618 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:56 smithi008 ceph-mon[21873]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 623 op/s 2024-04-27T02:12:56.618 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:56 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.618 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:56 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 71470 18.60 MB/sec execute 3 sec latency 38.850 ms 2024-04-27T02:12:56.863 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:56 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.863 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:56 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.864 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:56 smithi077 ceph-mon[24318]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 623 op/s 2024-04-27T02:12:56.864 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:56 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:56.864 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:56 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:57.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:57 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:12:57.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72058 15.86 MB/sec execute 4 sec latency 33.594 ms 2024-04-27T02:12:57.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:57 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:12:58.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 72505 14.79 MB/sec execute 5 sec latency 34.277 ms 2024-04-27T02:12:59.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 623 op/s 2024-04-27T02:12:59.143 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi008 to 1937M 2024-04-27T02:12:59.144 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:12:59 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 623 op/s 2024-04-27T02:12:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:12:59.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi008 to 1937M 2024-04-27T02:12:59.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:12:59 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:12:59.317 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:12:59.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 73169 15.24 MB/sec execute 6 sec latency 34.123 ms 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 3s ago 8m 26.1M - 0.23.0 ba2b418f427c aa5e9072673c 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 3s ago 8m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (7m) 5s ago 7m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (6m) 3s ago 8m 53.7M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (2m) 3s ago 2m 39.4M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (2m) 5s ago 2m 12.8M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (9m) 3s ago 9m 304M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (29s) 5s ago 7m 492M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 b9fb6e7160ad 2024-04-27T02:13:00.338 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 3s ago 9m 48.8M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (7m) 5s ago 7m 39.5M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (2m) 3s ago 2m 241M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (2m) 5s ago 2m 76.0M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 3s ago 8m 19.4M - 1.3.1 1dbe0e931976 3563103131a3 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (7m) 5s ago 7m 17.7M - 1.3.1 1dbe0e931976 d9d76f2322b0 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 3s ago 7m 296M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 3s ago 6m 226M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 3s ago 6m 183M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 3s ago 5m 231M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (5m) 5s ago 5m 245M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (4m) 5s ago 4m 278M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (4m) 5s ago 4m 247M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:13:00.339 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (4m) 5s ago 4m 340M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:13:00.340 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 3s ago 7m 63.9M - 2.33.4 514e6a882f6e ce98632a0fef 2024-04-27T02:13:00.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 73729 14.97 MB/sec execute 7 sec latency 30.620 ms 2024-04-27T02:13:00.918 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:13:00.918 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:13:00.918 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:13:00.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:13:00.919 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:13:00.920 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='client.24483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: pgmap v12: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 9.0 MiB/s wr, 962 op/s 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.170 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: Adjusting osd_memory_target on smithi077 to 2704M 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:13:01.171 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:01 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1216222669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:13:01.461 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:13:01.461 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:13:01.461 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:13:01.461 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:13:01.462 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:13:01.462 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T02:13:01.462 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-27T02:13:01.462 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:13:01.462 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:13:01.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='client.24483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:01.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:01.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: pgmap v12: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 9.0 MiB/s wr, 962 op/s 2024-04-27T02:13:01.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: Adjusting osd_memory_target on smithi077 to 2704M 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:13:01.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:01 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1216222669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:13:01.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74187 15.41 MB/sec execute 8 sec latency 35.521 ms 2024-04-27T02:13:02.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:02 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.238 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:13:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:13:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:02 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:02.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 74815 16.26 MB/sec execute 9 sec latency 27.755 ms 2024-04-27T02:13:03.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:03 smithi008 ceph-mon[21873]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:03.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:03 smithi008 ceph-mon[21873]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-27T02:13:03.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:03 smithi008 ceph-mon[21873]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-27T02:13:03.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:03 smithi008 ceph-mon[21873]: pgmap v13: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 611 op/s 2024-04-27T02:13:03.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:03 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/127975589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:13:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:03 smithi077 ceph-mon[24318]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:03 smithi077 ceph-mon[24318]: Reconfiguring alertmanager.smithi008 (dependencies changed)... 2024-04-27T02:13:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:03 smithi077 ceph-mon[24318]: Reconfiguring daemon alertmanager.smithi008 on smithi008 2024-04-27T02:13:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:03 smithi077 ceph-mon[24318]: pgmap v13: 97 pgs: 97 active+clean; 342 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 611 op/s 2024-04-27T02:13:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:03 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/127975589' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:13:03.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75187 15.53 MB/sec execute 10 sec latency 29.439 ms 2024-04-27T02:13:04.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 75617 14.46 MB/sec execute 11 sec latency 29.070 ms 2024-04-27T02:13:05.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:05 smithi008 ceph-mon[21873]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 924 op/s 2024-04-27T02:13:05.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:05 smithi077 ceph-mon[24318]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 924 op/s 2024-04-27T02:13:05.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 76036 14.10 MB/sec execute 12 sec latency 32.195 ms 2024-04-27T02:13:06.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 76776 14.43 MB/sec execute 13 sec latency 44.563 ms 2024-04-27T02:13:07.228 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:07 smithi008 ceph-mon[21873]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.0 MiB/s wr, 651 op/s 2024-04-27T02:13:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:07 smithi077 ceph-mon[24318]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.0 MiB/s wr, 651 op/s 2024-04-27T02:13:07.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77334 14.47 MB/sec execute 14 sec latency 34.629 ms 2024-04-27T02:13:08.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 77957 15.12 MB/sec execute 15 sec latency 36.364 ms 2024-04-27T02:13:09.210 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:09 smithi008 ceph-mon[21873]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.0 MiB/s wr, 651 op/s 2024-04-27T02:13:09.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:09 smithi077 ceph-mon[24318]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.0 MiB/s wr, 651 op/s 2024-04-27T02:13:09.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78485 15.32 MB/sec execute 16 sec latency 30.533 ms 2024-04-27T02:13:10.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 78912 14.83 MB/sec execute 17 sec latency 25.680 ms 2024-04-27T02:13:11.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:10 smithi008 ceph-mon[21873]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 962 op/s 2024-04-27T02:13:11.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:10 smithi077 ceph-mon[24318]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 962 op/s 2024-04-27T02:13:11.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79362 14.34 MB/sec execute 18 sec latency 34.583 ms 2024-04-27T02:13:12.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 79807 14.31 MB/sec execute 19 sec latency 43.763 ms 2024-04-27T02:13:13.310 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:13 smithi008 ceph-mon[21873]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 623 op/s 2024-04-27T02:13:13.311 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:13 smithi077 ceph-mon[24318]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 623 op/s 2024-04-27T02:13:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:13.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 80440 14.21 MB/sec execute 20 sec latency 34.013 ms 2024-04-27T02:13:14.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 80909 14.25 MB/sec execute 21 sec latency 43.884 ms 2024-04-27T02:13:15.334 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:15 smithi008 ceph-mon[21873]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.4 MiB/s wr, 929 op/s 2024-04-27T02:13:15.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:15 smithi077 ceph-mon[24318]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.4 MiB/s wr, 929 op/s 2024-04-27T02:13:15.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 81493 14.66 MB/sec execute 22 sec latency 31.378 ms 2024-04-27T02:13:16.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 81913 14.61 MB/sec execute 23 sec latency 36.699 ms 2024-04-27T02:13:17.381 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:17 smithi008 ceph-mon[21873]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 616 op/s 2024-04-27T02:13:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:17 smithi077 ceph-mon[24318]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 616 op/s 2024-04-27T02:13:17.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82280 14.45 MB/sec execute 24 sec latency 29.990 ms 2024-04-27T02:13:18.846 INFO:teuthology.orchestra.run.smithi008.stdout: 5 82715 14.02 MB/sec execute 25 sec latency 49.386 ms 2024-04-27T02:13:19.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:19 smithi008 ceph-mon[21873]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 616 op/s 2024-04-27T02:13:19.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:19 smithi077 ceph-mon[24318]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 616 op/s 2024-04-27T02:13:19.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83320 14.12 MB/sec execute 26 sec latency 21.210 ms 2024-04-27T02:13:20.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 83955 14.07 MB/sec execute 27 sec latency 39.929 ms 2024-04-27T02:13:20.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-27T02:13:20.914 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:20 smithi008 ceph-mon[21873]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 899 op/s 2024-04-27T02:13:20.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: Reconfiguring grafana.smithi008 (dependencies changed)... 2024-04-27T02:13:20.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: Reconfiguring daemon grafana.smithi008 on smithi008 2024-04-27T02:13:20.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:20 smithi077 ceph-mon[24318]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 899 op/s 2024-04-27T02:13:21.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 84578 14.27 MB/sec execute 28 sec latency 27.281 ms 2024-04-27T02:13:22.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85203 14.60 MB/sec execute 29 sec latency 23.574 ms 2024-04-27T02:13:23.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:23 smithi008 ceph-mon[21873]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-04-27T02:13:23.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:23 smithi077 ceph-mon[24318]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-04-27T02:13:23.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 85826 14.74 MB/sec execute 30 sec latency 32.232 ms 2024-04-27T02:13:24.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 86342 14.42 MB/sec execute 31 sec latency 34.773 ms 2024-04-27T02:13:25.190 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:25 smithi008 ceph-mon[21873]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 939 op/s 2024-04-27T02:13:25.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:25 smithi077 ceph-mon[24318]: pgmap v24: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 939 op/s 2024-04-27T02:13:25.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 86849 14.46 MB/sec execute 32 sec latency 45.416 ms 2024-04-27T02:13:26.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87467 14.38 MB/sec execute 33 sec latency 51.151 ms 2024-04-27T02:13:27.215 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:27 smithi008 ceph-mon[21873]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-27T02:13:27.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:27 smithi077 ceph-mon[24318]: pgmap v25: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-27T02:13:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 87765 14.19 MB/sec execute 34 sec latency 42.141 ms 2024-04-27T02:13:28.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:28 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:28.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:28 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:28.847 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88234 14.29 MB/sec execute 35 sec latency 29.390 ms 2024-04-27T02:13:29.230 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:29 smithi008 ceph-mon[21873]: pgmap v26: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-27T02:13:29.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:29 smithi077 ceph-mon[24318]: pgmap v26: 97 pgs: 97 active+clean; 341 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-27T02:13:29.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 88621 14.29 MB/sec execute 36 sec latency 33.481 ms 2024-04-27T02:13:30.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89028 14.26 MB/sec execute 37 sec latency 31.447 ms 2024-04-27T02:13:30.983 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:30 smithi008 ceph-mon[21873]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 934 op/s 2024-04-27T02:13:31.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:30 smithi077 ceph-mon[24318]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 934 op/s 2024-04-27T02:13:31.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89363 14.15 MB/sec execute 38 sec latency 28.704 ms 2024-04-27T02:13:32.844 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:13:32.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 89804 13.88 MB/sec execute 39 sec latency 38.549 ms 2024-04-27T02:13:33.248 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:33 smithi008 ceph-mon[21873]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.3 MiB/s wr, 651 op/s 2024-04-27T02:13:33.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:33 smithi077 ceph-mon[24318]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.3 MiB/s wr, 651 op/s 2024-04-27T02:13:33.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 90248 13.83 MB/sec execute 40 sec latency 39.327 ms 2024-04-27T02:13:34.003 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:13:34.003 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 37s ago 9m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (8m) 39s ago 8m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (7m) 37s ago 8m 53.7M - 8.3.5 dad864ee21e9 37cfcc6d6fb4 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (3m) 37s ago 3m 39.4M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (3m) 39s ago 3m 12.8M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (10m) 37s ago 10m 304M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (63s) 39s ago 8m 492M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 b9fb6e7160ad 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 37s ago 10m 48.8M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (8m) 39s ago 8m 39.5M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (2m) 37s ago 2m 241M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (2m) 39s ago 2m 76.0M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 37s ago 8m 19.4M - 1.3.1 1dbe0e931976 3563103131a3 2024-04-27T02:13:34.004 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (8m) 39s ago 8m 17.7M - 1.3.1 1dbe0e931976 d9d76f2322b0 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 37s ago 7m 296M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 37s ago 7m 226M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 37s ago 6m 183M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 37s ago 6m 231M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (5m) 39s ago 5m 245M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (5m) 39s ago 5m 278M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (5m) 39s ago 5m 247M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (4m) 39s ago 4m 340M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:13:34.005 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 37s ago 7m 63.9M - 2.33.4 514e6a882f6e ce98632a0fef 2024-04-27T02:13:34.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:34 smithi008 ceph-mon[21873]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:34.469 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:34 smithi077 ceph-mon[24318]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:34.729 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:13:34.730 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:13:34.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 90791 13.83 MB/sec execute 41 sec latency 30.884 ms 2024-04-27T02:13:35.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:35 smithi008 ceph-mon[21873]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:35.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:35 smithi008 ceph-mon[21873]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:35.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:35 smithi008 ceph-mon[21873]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 897 op/s 2024-04-27T02:13:35.263 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:35 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1052034874' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:13:35.343 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:13:35.344 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:13:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:35 smithi077 ceph-mon[24318]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:35 smithi077 ceph-mon[24318]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:35 smithi077 ceph-mon[24318]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 897 op/s 2024-04-27T02:13:35.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:35 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1052034874' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:13:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91348 13.78 MB/sec execute 42 sec latency 28.024 ms 2024-04-27T02:13:36.066 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:13:36.271 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:36 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3493661285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:13:36.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:36 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3493661285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:13:36.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 91881 13.92 MB/sec execute 43 sec latency 25.691 ms 2024-04-27T02:13:37.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:37 smithi008 ceph-mon[21873]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:37.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:37 smithi008 ceph-mon[21873]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:13:37.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:37 smithi077 ceph-mon[24318]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:13:37.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:37 smithi077 ceph-mon[24318]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:13:37.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92498 14.13 MB/sec execute 44 sec latency 24.958 ms 2024-04-27T02:13:38.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 92942 14.08 MB/sec execute 45 sec latency 23.987 ms 2024-04-27T02:13:39.316 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:39 smithi008 ceph-mon[21873]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:13:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:39 smithi077 ceph-mon[24318]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:13:39.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93400 13.87 MB/sec execute 46 sec latency 36.134 ms 2024-04-27T02:13:40.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 93841 13.82 MB/sec execute 47 sec latency 30.105 ms 2024-04-27T02:13:41.093 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:40 smithi008 ceph-mon[21873]: pgmap v32: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 863 op/s 2024-04-27T02:13:41.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:40 smithi077 ceph-mon[24318]: pgmap v32: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 863 op/s 2024-04-27T02:13:41.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94441 13.83 MB/sec execute 48 sec latency 40.779 ms 2024-04-27T02:13:42.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 94879 13.76 MB/sec execute 49 sec latency 39.943 ms 2024-04-27T02:13:43.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:43 smithi008 ceph-mon[21873]: pgmap v33: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.2 MiB/s wr, 562 op/s 2024-04-27T02:13:43.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:43 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:43.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:43 smithi077 ceph-mon[24318]: pgmap v33: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.2 MiB/s wr, 562 op/s 2024-04-27T02:13:43.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:43 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:43.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95284 13.81 MB/sec execute 50 sec latency 36.540 ms 2024-04-27T02:13:44.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:44.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:44.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:44 smithi008 ceph-mon[21873]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-04-27T02:13:44.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:44 smithi008 ceph-mon[21873]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-04-27T02:13:44.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:44.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:44.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:44 smithi077 ceph-mon[24318]: Reconfiguring node-exporter.smithi008 (dependencies changed)... 2024-04-27T02:13:44.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:44 smithi077 ceph-mon[24318]: Reconfiguring daemon node-exporter.smithi008 on smithi008 2024-04-27T02:13:44.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 95814 13.91 MB/sec execute 51 sec latency 54.488 ms 2024-04-27T02:13:45.422 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:45 smithi008 ceph-mon[21873]: pgmap v34: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 855 op/s 2024-04-27T02:13:45.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:45 smithi077 ceph-mon[24318]: pgmap v34: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 855 op/s 2024-04-27T02:13:45.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96215 13.92 MB/sec execute 52 sec latency 27.246 ms 2024-04-27T02:13:46.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 96615 13.77 MB/sec execute 53 sec latency 59.862 ms 2024-04-27T02:13:47.438 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:47 smithi008 ceph-mon[21873]: pgmap v35: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T02:13:47.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:47 smithi077 ceph-mon[24318]: pgmap v35: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T02:13:47.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97042 13.62 MB/sec execute 54 sec latency 52.171 ms 2024-04-27T02:13:48.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:48 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:48.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:48 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:48.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:48 smithi008 ceph-mon[21873]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-27T02:13:48.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:48 smithi008 ceph-mon[21873]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-27T02:13:48.448 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:48 smithi008 ceph-mon[21873]: pgmap v36: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T02:13:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:48 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:48 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:13:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:48 smithi077 ceph-mon[24318]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-27T02:13:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:48 smithi077 ceph-mon[24318]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-27T02:13:48.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:48 smithi077 ceph-mon[24318]: pgmap v36: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T02:13:48.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97372 13.55 MB/sec execute 55 sec latency 53.569 ms 2024-04-27T02:13:49.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 97890 13.51 MB/sec execute 56 sec latency 43.612 ms 2024-04-27T02:13:50.848 INFO:teuthology.orchestra.run.smithi008.stdout: 5 98352 13.49 MB/sec execute 57 sec latency 35.255 ms 2024-04-27T02:13:50.988 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:50 smithi008 ceph-mon[21873]: pgmap v37: 97 pgs: 97 active+clean; 343 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 869 op/s 2024-04-27T02:13:51.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:50 smithi077 ceph-mon[24318]: pgmap v37: 97 pgs: 97 active+clean; 343 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 869 op/s 2024-04-27T02:13:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 98711 13.48 MB/sec execute 58 sec latency 39.098 ms 2024-04-27T02:13:52.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99268 13.56 MB/sec execute 59 sec latency 27.464 ms 2024-04-27T02:13:53.267 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:53 smithi008 ceph-mon[21873]: pgmap v38: 97 pgs: 97 active+clean; 343 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-04-27T02:13:53.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:53 smithi077 ceph-mon[24318]: pgmap v38: 97 pgs: 97 active+clean; 343 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-04-27T02:13:53.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99638 13.57 MB/sec execute 60 sec latency 44.926 ms 2024-04-27T02:13:54.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 99899 13.45 MB/sec execute 61 sec latency 65.046 ms 2024-04-27T02:13:55.295 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:55 smithi008 ceph-mon[21873]: pgmap v39: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 814 op/s 2024-04-27T02:13:55.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:55 smithi077 ceph-mon[24318]: pgmap v39: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 814 op/s 2024-04-27T02:13:55.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100169 13.32 MB/sec execute 62 sec latency 50.985 ms 2024-04-27T02:13:56.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100468 13.16 MB/sec execute 63 sec latency 52.984 ms 2024-04-27T02:13:57.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:57 smithi008 ceph-mon[21873]: pgmap v40: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 521 op/s 2024-04-27T02:13:57.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:57 smithi077 ceph-mon[24318]: pgmap v40: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 521 op/s 2024-04-27T02:13:57.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 100802 13.08 MB/sec execute 64 sec latency 37.586 ms 2024-04-27T02:13:58.231 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:58 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:58.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:58 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:13:58.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101264 13.08 MB/sec execute 65 sec latency 33.988 ms 2024-04-27T02:13:59.259 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:13:59 smithi008 ceph-mon[21873]: pgmap v41: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 521 op/s 2024-04-27T02:13:59.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:13:59 smithi077 ceph-mon[24318]: pgmap v41: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 521 op/s 2024-04-27T02:13:59.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 101753 13.02 MB/sec execute 66 sec latency 38.718 ms 2024-04-27T02:14:00.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102179 13.03 MB/sec execute 67 sec latency 49.160 ms 2024-04-27T02:14:01.060 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:00 smithi008 ceph-mon[21873]: pgmap v42: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 733 op/s 2024-04-27T02:14:01.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:00 smithi077 ceph-mon[24318]: pgmap v42: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 733 op/s 2024-04-27T02:14:01.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 102598 13.06 MB/sec execute 68 sec latency 42.338 ms 2024-04-27T02:14:02.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103094 13.15 MB/sec execute 69 sec latency 36.143 ms 2024-04-27T02:14:03.244 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:03 smithi008 ceph-mon[21873]: pgmap v43: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-04-27T02:14:03.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:03 smithi077 ceph-mon[24318]: pgmap v43: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-04-27T02:14:03.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103353 13.08 MB/sec execute 70 sec latency 149.667 ms 2024-04-27T02:14:04.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 103738 12.98 MB/sec execute 71 sec latency 34.618 ms 2024-04-27T02:14:05.278 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:05 smithi008 ceph-mon[21873]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 716 op/s 2024-04-27T02:14:05.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:05 smithi077 ceph-mon[24318]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 716 op/s 2024-04-27T02:14:05.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104132 12.86 MB/sec execute 72 sec latency 39.373 ms 2024-04-27T02:14:06.658 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:14:06.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104472 12.83 MB/sec execute 73 sec latency 49.710 ms 2024-04-27T02:14:07.297 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:07 smithi008 ceph-mon[21873]: pgmap v45: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 454 op/s 2024-04-27T02:14:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:07 smithi077 ceph-mon[24318]: pgmap v45: 97 pgs: 97 active+clean; 339 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 454 op/s 2024-04-27T02:14:07.750 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 starting - - - - 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 71s ago 9m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (9m) 73s ago 8m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 starting - - - - 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (3m) 71s ago 3m 39.4M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (3m) 73s ago 3m 12.8M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:9283 running (10m) 71s ago 10m 304M - 17.2.0 e1d6a67b021e 59da5e447cb5 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (96s) 73s ago 8m 492M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 b9fb6e7160ad 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 71s ago 11m 48.8M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (8m) 73s ago 8m 39.5M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (3m) 71s ago 3m 241M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (3m) 73s ago 3m 76.0M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 starting - - - - 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (8m) 73s ago 8m 17.7M - 1.3.1 1dbe0e931976 d9d76f2322b0 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 71s ago 8m 296M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:14:07.751 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (7m) 71s ago 7m 226M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 71s ago 7m 183M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 71s ago 6m 231M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (6m) 73s ago 6m 245M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (5m) 73s ago 5m 278M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (5m) 73s ago 5m 247M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (5m) 73s ago 5m 340M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:14:07.752 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 71s ago 8m 63.9M - 2.33.4 514e6a882f6e ce98632a0fef 2024-04-27T02:14:07.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 104945 12.79 MB/sec execute 74 sec latency 48.671 ms 2024-04-27T02:14:08.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:08 smithi008 ceph-mon[21873]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:08.304 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:08 smithi008 ceph-mon[21873]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:08.457 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:14:08.458 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:14:08.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:08 smithi077 ceph-mon[24318]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:08.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:08 smithi077 ceph-mon[24318]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:08.849 INFO:teuthology.orchestra.run.smithi008.stdout: 5 105304 12.74 MB/sec execute 75 sec latency 87.766 ms 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:14:08.977 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:14:09.339 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:09 smithi008 ceph-mon[21873]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:09.339 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:09 smithi008 ceph-mon[21873]: pgmap v46: 97 pgs: 97 active+clean; 336 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 471 op/s 2024-04-27T02:14:09.339 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:09 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1315755820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:14:09.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:09 smithi077 ceph-mon[24318]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:09.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:09 smithi077 ceph-mon[24318]: pgmap v46: 97 pgs: 97 active+clean; 336 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 471 op/s 2024-04-27T02:14:09.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:09 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1315755820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:14:09.712 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:14:09.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 105682 12.73 MB/sec execute 76 sec latency 28.514 ms 2024-04-27T02:14:10.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:10 smithi008 ceph-mon[21873]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:10.352 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:10 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/721971134' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:14:10.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:10 smithi077 ceph-mon[24318]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:10.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:10 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/721971134' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:14:10.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106055 12.75 MB/sec execute 77 sec latency 30.310 ms 2024-04-27T02:14:11.362 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:11 smithi008 ceph-mon[21873]: pgmap v47: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-27T02:14:11.362 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:11 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:11.363 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:11 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:11 smithi077 ceph-mon[24318]: pgmap v47: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-04-27T02:14:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:11 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:11.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:11 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:11.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106471 12.79 MB/sec execute 78 sec latency 37.324 ms 2024-04-27T02:14:12.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:12 smithi008 ceph-mon[21873]: Reconfiguring node-exporter.smithi077 (dependencies changed)... 2024-04-27T02:14:12.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:12 smithi008 ceph-mon[21873]: Reconfiguring daemon node-exporter.smithi077 on smithi077 2024-04-27T02:14:12.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:12 smithi077 ceph-mon[24318]: Reconfiguring node-exporter.smithi077 (dependencies changed)... 2024-04-27T02:14:12.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:12 smithi077 ceph-mon[24318]: Reconfiguring daemon node-exporter.smithi077 on smithi077 2024-04-27T02:14:12.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 106887 12.80 MB/sec execute 79 sec latency 33.808 ms 2024-04-27T02:14:13.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: pgmap v48: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 490 op/s 2024-04-27T02:14:13.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:13.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.385 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:13 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: pgmap v48: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 490 op/s 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:13.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:13 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:13.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107229 12.71 MB/sec execute 80 sec latency 43.613 ms 2024-04-27T02:14:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:14:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T02:14:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:14:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T02:14:14.397 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:14:14.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T02:14:14.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:14.398 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:14 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T02:14:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:14.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:14 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:14.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 107656 12.60 MB/sec execute 81 sec latency 34.002 ms 2024-04-27T02:14:15.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:15 smithi008 ceph-mon[21873]: pgmap v49: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.6 MiB/s wr, 720 op/s 2024-04-27T02:14:15.412 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:15 smithi008 ceph-mon[21873]: mgrmap e28: smithi077.snksrp(active, since 93s), standbys: smithi008.exkhsc 2024-04-27T02:14:15.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:15 smithi077 ceph-mon[24318]: pgmap v49: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.6 MiB/s wr, 720 op/s 2024-04-27T02:14:15.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:15 smithi077 ceph-mon[24318]: mgrmap e28: smithi077.snksrp(active, since 93s), standbys: smithi008.exkhsc 2024-04-27T02:14:15.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 108053 12.59 MB/sec execute 82 sec latency 48.526 ms 2024-04-27T02:14:16.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 108682 12.62 MB/sec execute 83 sec latency 30.850 ms 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: pgmap v50: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: Upgrade: Updating mgr.smithi008.exkhsc 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:14:17.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:17 smithi008 ceph-mon[21873]: Deploying daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: pgmap v50: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: Upgrade: Updating mgr.smithi008.exkhsc 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:14:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:17 smithi077 ceph-mon[24318]: Deploying daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:14:17.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109258 12.66 MB/sec execute 84 sec latency 31.852 ms 2024-04-27T02:14:18.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 109864 12.80 MB/sec execute 85 sec latency 24.510 ms 2024-04-27T02:14:19.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:19 smithi008 ceph-mon[21873]: pgmap v51: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 502 op/s 2024-04-27T02:14:19.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:19 smithi077 ceph-mon[24318]: pgmap v51: 97 pgs: 97 active+clean; 337 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 502 op/s 2024-04-27T02:14:19.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110252 12.78 MB/sec execute 86 sec latency 34.628 ms 2024-04-27T02:14:20.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110533 12.73 MB/sec execute 87 sec latency 46.547 ms 2024-04-27T02:14:20.978 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:20 smithi008 ceph-mon[21873]: pgmap v52: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 758 op/s 2024-04-27T02:14:21.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:20 smithi077 ceph-mon[24318]: pgmap v52: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 758 op/s 2024-04-27T02:14:21.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 110782 12.64 MB/sec execute 88 sec latency 138.874 ms 2024-04-27T02:14:22.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111116 12.54 MB/sec execute 89 sec latency 227.734 ms 2024-04-27T02:14:23.268 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:23 smithi008 ceph-mon[21873]: pgmap v53: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-04-27T02:14:23.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:23 smithi077 ceph-mon[24318]: pgmap v53: 97 pgs: 97 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-04-27T02:14:23.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111384 12.48 MB/sec execute 90 sec latency 50.817 ms 2024-04-27T02:14:24.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:24 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:24 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:24 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:24.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:24 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.285 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:24 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:24 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:24 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:24 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:24.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:24 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:24 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:24.850 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111602 12.37 MB/sec execute 91 sec latency 74.018 ms 2024-04-27T02:14:25.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:25 smithi008 ceph-mon[21873]: pgmap v54: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 745 op/s 2024-04-27T02:14:25.698 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:25 smithi077 ceph-mon[24318]: pgmap v54: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 745 op/s 2024-04-27T02:14:25.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 111961 12.37 MB/sec execute 92 sec latency 62.169 ms 2024-04-27T02:14:26.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112361 12.32 MB/sec execute 93 sec latency 50.869 ms 2024-04-27T02:14:27.282 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:27 smithi008 ceph-mon[21873]: pgmap v55: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-04-27T02:14:27.282 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:27 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:27.601 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:27 smithi077 ceph-mon[24318]: pgmap v55: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-04-27T02:14:27.601 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:27 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 112713 12.31 MB/sec execute 94 sec latency 61.709 ms 2024-04-27T02:14:28.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113048 12.31 MB/sec execute 95 sec latency 53.213 ms 2024-04-27T02:14:29.321 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:29 smithi008 ceph-mon[21873]: pgmap v56: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-04-27T02:14:29.321 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:29 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:29.321 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:29 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:29.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:29 smithi077 ceph-mon[24318]: pgmap v56: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 515 op/s 2024-04-27T02:14:29.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:29 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:29.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:29 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:29.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113356 12.29 MB/sec execute 96 sec latency 47.192 ms 2024-04-27T02:14:30.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113737 12.31 MB/sec execute 97 sec latency 49.210 ms 2024-04-27T02:14:31.095 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:30 smithi008 ceph-mon[21873]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 681 op/s 2024-04-27T02:14:31.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:30 smithi077 ceph-mon[24318]: pgmap v57: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 681 op/s 2024-04-27T02:14:31.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 113989 12.27 MB/sec execute 98 sec latency 58.821 ms 2024-04-27T02:14:32.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114237 12.20 MB/sec execute 99 sec latency 74.719 ms 2024-04-27T02:14:33.153 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:33 smithi008 ceph-mon[21873]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 409 op/s 2024-04-27T02:14:33.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:33 smithi077 ceph-mon[24318]: pgmap v58: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.0 MiB/s wr, 409 op/s 2024-04-27T02:14:33.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114569 12.11 MB/sec execute 100 sec latency 47.175 ms 2024-04-27T02:14:34.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: Standby manager daemon smithi008.exkhsc restarted 2024-04-27T02:14:34.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:14:34.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/crt"}]: dispatch 2024-04-27T02:14:34.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:14:34.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/key"}]: dispatch 2024-04-27T02:14:34.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:14:34.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:34.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:34 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: Standby manager daemon smithi008.exkhsc restarted 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: Standby manager daemon smithi008.exkhsc started 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/crt"}]: dispatch 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi008.exkhsc/key"}]: dispatch 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:14:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:34.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:34 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:34.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 114903 12.05 MB/sec execute 101 sec latency 42.652 ms 2024-04-27T02:14:35.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:35 smithi008 ceph-mon[21873]: pgmap v59: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-04-27T02:14:35.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:35 smithi008 ceph-mon[21873]: mgrmap e29: smithi077.snksrp(active, since 113s), standbys: smithi008.exkhsc 2024-04-27T02:14:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:35 smithi077 ceph-mon[24318]: pgmap v59: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-04-27T02:14:35.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:35 smithi077 ceph-mon[24318]: mgrmap e29: smithi077.snksrp(active, since 113s), standbys: smithi008.exkhsc 2024-04-27T02:14:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 115402 12.08 MB/sec execute 102 sec latency 49.066 ms 2024-04-27T02:14:36.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:36 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:36.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:36 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:36.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:36 smithi008 ceph-mon[21873]: pgmap v60: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 378 op/s 2024-04-27T02:14:36.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:36 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:36.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:36 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:36.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:36 smithi077 ceph-mon[24318]: pgmap v60: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 378 op/s 2024-04-27T02:14:36.853 INFO:teuthology.orchestra.run.smithi008.stdout: 5 116085 12.13 MB/sec execute 103 sec latency 23.362 ms 2024-04-27T02:14:37.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 116594 12.16 MB/sec execute 104 sec latency 49.725 ms 2024-04-27T02:14:38.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117078 12.20 MB/sec execute 105 sec latency 46.698 ms 2024-04-27T02:14:39.317 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:39 smithi008 ceph-mon[21873]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 398 op/s 2024-04-27T02:14:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:39 smithi077 ceph-mon[24318]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 398 op/s 2024-04-27T02:14:39.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117519 12.24 MB/sec execute 106 sec latency 38.220 ms 2024-04-27T02:14:40.340 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:14:40.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 117810 12.18 MB/sec execute 107 sec latency 64.944 ms 2024-04-27T02:14:41.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:40 smithi008 ceph-mon[21873]: pgmap v62: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 668 op/s 2024-04-27T02:14:41.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:40 smithi077 ceph-mon[24318]: pgmap v62: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.8 MiB/s wr, 668 op/s 2024-04-27T02:14:41.421 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:14:41.421 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (83s) 7s ago 10m 17.7M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:14:41.421 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (10m) 7s ago 10m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (9m) 13s ago 9m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 unknown 7s ago 9m - - 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (4m) 7s ago 4m 39.3M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (4m) 13s ago 4m 13.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (19s) 7s ago 11m 164M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (2m) 13s ago 9m 510M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 b9fb6e7160ad 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 7s ago 11m 51.2M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (9m) 13s ago 9m 40.3M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (4m) 7s ago 4m 265M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (3m) 13s ago 3m 83.8M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (54s) 7s ago 9m 9.99M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (29s) 13s ago 9m 9.82M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 7s ago 8m 442M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:14:41.422 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 7s ago 8m 327M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 7s ago 7m 285M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 7s ago 7m 399M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (6m) 13s ago 6m 383M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (6m) 13s ago 6m 416M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (6m) 13s ago 6m 359M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (5m) 13s ago 5m 498M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:14:41.423 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (31s) 7s ago 9m 53.4M - 2.33.4 514e6a882f6e 7479c7ec023a 2024-04-27T02:14:41.851 INFO:teuthology.orchestra.run.smithi008.stdout: 5 118101 12.09 MB/sec execute 108 sec latency 45.954 ms 2024-04-27T02:14:42.011 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:14:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:14:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:14:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:14:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-27T02:14:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:14:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:14:42.013 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:14:42.108 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:41 smithi008 ceph-mon[21873]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:42.108 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:41 smithi008 ceph-mon[21873]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:42.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:41 smithi077 ceph-mon[24318]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:42.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:41 smithi077 ceph-mon[24318]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T02:14:42.534 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mgr daemons", 2024-04-27T02:14:42.535 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:14:42.535 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:14:42.854 INFO:teuthology.orchestra.run.smithi008.stdout: 5 118409 12.03 MB/sec execute 109 sec latency 51.877 ms 2024-04-27T02:14:43.066 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:14:43.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:42 smithi008 ceph-mon[21873]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:43.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:42 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/2269909561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:14:43.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:42 smithi008 ceph-mon[21873]: pgmap v63: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.0 MiB/s wr, 476 op/s 2024-04-27T02:14:43.146 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:42 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:43.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:42 smithi077 ceph-mon[24318]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:43.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:42 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/2269909561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:14:43.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:42 smithi077 ceph-mon[24318]: pgmap v63: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.0 MiB/s wr, 476 op/s 2024-04-27T02:14:43.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:42 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:43.854 INFO:teuthology.orchestra.run.smithi008.stdout: 5 118918 12.06 MB/sec execute 110 sec latency 70.027 ms 2024-04-27T02:14:44.168 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:43 smithi008 ceph-mon[21873]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:44.168 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:43 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/3658614121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:14:44.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:43 smithi077 ceph-mon[24318]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:14:44.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:43 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/3658614121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:14:44.854 INFO:teuthology.orchestra.run.smithi008.stdout: 5 119477 12.10 MB/sec execute 111 sec latency 45.121 ms 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: pgmap v64: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 693 op/s 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:45.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:45.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:44 smithi008 ceph-mon[21873]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: pgmap v64: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 693 op/s 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 172.21.15.77:0/3083837439' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:44 smithi077 ceph-mon[24318]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T02:14:45.854 INFO:teuthology.orchestra.run.smithi008.stdout: 5 119973 12.08 MB/sec execute 112 sec latency 37.048 ms 2024-04-27T02:14:46.181 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: pgmap v65: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.8 MiB/s wr, 590 op/s 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: Failing over to other MGR 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd='[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]': finished 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: mgrmap e30: smithi008.exkhsc(active, starting, since 0.859959s) 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:14:46.182 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:46.183 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:46.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:45 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:14:46.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: pgmap v65: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.8 MiB/s wr, 590 op/s 2024-04-27T02:14:46.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: Upgrade: Need to upgrade myself (mgr.smithi077.snksrp) 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: Failing over to other MGR 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14674 ' entity='mgr.smithi077.snksrp' cmd='[{"prefix": "mgr fail", "who": "smithi077.snksrp"}]': finished 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: mgrmap e30: smithi008.exkhsc(active, starting, since 0.859959s) 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:14:46.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:14:46.236 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:14:46.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:14:46.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:14:46.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:46.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:14:46.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:45 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:14:46.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120381 12.11 MB/sec execute 113 sec latency 45.339 ms 2024-04-27T02:14:47.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:47 smithi008 ceph-mon[21873]: mgrmap e31: smithi008.exkhsc(active, since 1.86537s) 2024-04-27T02:14:47.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:47 smithi008 ceph-mon[21873]: pgmap v3: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:47.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:47 smithi077 ceph-mon[24318]: mgrmap e31: smithi008.exkhsc(active, since 1.86537s) 2024-04-27T02:14:47.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:47 smithi077 ceph-mon[24318]: pgmap v3: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:47.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 120936 12.17 MB/sec execute 114 sec latency 28.796 ms 2024-04-27T02:14:48.482 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:48 smithi008 ceph-mon[21873]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:14:48.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:48 smithi008 ceph-mon[21873]: Cluster is now healthy 2024-04-27T02:14:48.483 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:48 smithi008 ceph-mon[21873]: pgmap v4: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:48.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:48 smithi077 ceph-mon[24318]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:14:48.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:48 smithi077 ceph-mon[24318]: Cluster is now healthy 2024-04-27T02:14:48.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:48 smithi077 ceph-mon[24318]: pgmap v4: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:48.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121250 12.15 MB/sec execute 115 sec latency 52.821 ms 2024-04-27T02:14:49.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:49 smithi008 ceph-mon[21873]: mgrmap e32: smithi008.exkhsc(active, since 3s) 2024-04-27T02:14:49.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:49 smithi008 ceph-mon[21873]: [27/Apr/2024:02:14:48] ENGINE Bus STARTING 2024-04-27T02:14:49.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:49 smithi008 ceph-mon[21873]: [27/Apr/2024:02:14:48] ENGINE Serving on http://172.21.15.8:8765 2024-04-27T02:14:49.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:49 smithi008 ceph-mon[21873]: [27/Apr/2024:02:14:48] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:14:49.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:49 smithi008 ceph-mon[21873]: [27/Apr/2024:02:14:48] ENGINE Bus STARTED 2024-04-27T02:14:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:49 smithi077 ceph-mon[24318]: mgrmap e32: smithi008.exkhsc(active, since 3s) 2024-04-27T02:14:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:49 smithi077 ceph-mon[24318]: [27/Apr/2024:02:14:48] ENGINE Bus STARTING 2024-04-27T02:14:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:49 smithi077 ceph-mon[24318]: [27/Apr/2024:02:14:48] ENGINE Serving on http://172.21.15.8:8765 2024-04-27T02:14:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:49 smithi077 ceph-mon[24318]: [27/Apr/2024:02:14:48] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:14:49.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:49 smithi077 ceph-mon[24318]: [27/Apr/2024:02:14:48] ENGINE Bus STARTED 2024-04-27T02:14:49.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121501 12.08 MB/sec execute 116 sec latency 58.389 ms 2024-04-27T02:14:50.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 121843 12.02 MB/sec execute 117 sec latency 42.371 ms 2024-04-27T02:14:51.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:50 smithi008 ceph-mon[21873]: pgmap v5: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:51.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:50 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:51.033 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:50 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:51.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:50 smithi077 ceph-mon[24318]: pgmap v5: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:51.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:50 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:51.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:50 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 122217 11.96 MB/sec execute 118 sec latency 61.892 ms 2024-04-27T02:14:52.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 122915 12.08 MB/sec execute 119 sec latency 25.744 ms 2024-04-27T02:14:53.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:52 smithi008 ceph-mon[21873]: pgmap v6: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:53.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:52 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:53.149 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:52 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:53.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:52 smithi077 ceph-mon[24318]: pgmap v6: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:14:53.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:52 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:53.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:52 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:53.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123506 12.08 MB/sec execute 120 sec latency 28.892 ms 2024-04-27T02:14:54.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 123823 12.06 MB/sec execute 121 sec latency 45.244 ms 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 MiB/s wr, 400 op/s 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:14:55.179 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:55 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 3.9 MiB/s wr, 400 op/s 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:14:55.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:55 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2192818660' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:14:55.858 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124270 12.07 MB/sec execute 122 sec latency 43.122 ms 2024-04-27T02:14:56.186 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:56 smithi008 ceph-mon[21873]: mgrmap e33: smithi008.exkhsc(active, since 10s), standbys: smithi077.snksrp 2024-04-27T02:14:56.187 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:56 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:56 smithi077 ceph-mon[24318]: mgrmap e33: smithi008.exkhsc(active, since 10s), standbys: smithi077.snksrp 2024-04-27T02:14:56.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:56 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:14:56.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 124899 12.16 MB/sec execute 123 sec latency 46.335 ms 2024-04-27T02:14:57.184 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:57 smithi008 ceph-mon[21873]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-04-27T02:14:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:57 smithi077 ceph-mon[24318]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-04-27T02:14:57.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125493 12.12 MB/sec execute 124 sec latency 34.002 ms 2024-04-27T02:14:58.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:58 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:58.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:58 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:58.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:58 smithi008 ceph-mon[21873]: pgmap v9: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 594 op/s 2024-04-27T02:14:58.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:58 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:58.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:58 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:58.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:58 smithi077 ceph-mon[24318]: pgmap v9: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 594 op/s 2024-04-27T02:14:58.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 125806 12.06 MB/sec execute 125 sec latency 38.156 ms 2024-04-27T02:14:59.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126154 12.06 MB/sec execute 126 sec latency 50.005 ms 2024-04-27T02:14:59.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:59 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:14:59.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:14:59 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:00.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:59 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:00.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:14:59 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:00.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 126577 12.09 MB/sec execute 127 sec latency 63.449 ms 2024-04-27T02:15:01.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:00 smithi008 ceph-mon[21873]: pgmap v10: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-04-27T02:15:01.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:00 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:01.015 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:00 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:01.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:00 smithi077 ceph-mon[24318]: pgmap v10: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-04-27T02:15:01.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:00 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:01.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:00 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:01.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127003 12.04 MB/sec execute 128 sec latency 35.414 ms 2024-04-27T02:15:02.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127255 12.03 MB/sec execute 129 sec latency 43.962 ms 2024-04-27T02:15:03.040 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:02 smithi008 ceph-mon[21873]: pgmap v11: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 554 op/s 2024-04-27T02:15:03.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:02 smithi077 ceph-mon[24318]: pgmap v11: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 5.9 MiB/s wr, 554 op/s 2024-04-27T02:15:03.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127603 12.03 MB/sec execute 130 sec latency 53.542 ms 2024-04-27T02:15:04.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 127926 12.01 MB/sec execute 131 sec latency 103.087 ms 2024-04-27T02:15:05.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:04 smithi008 ceph-mon[21873]: pgmap v12: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 811 op/s 2024-04-27T02:15:05.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:04 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:05.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:04 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:05.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:04 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:05.071 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:04 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:15:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:04 smithi077 ceph-mon[24318]: pgmap v12: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 811 op/s 2024-04-27T02:15:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:04 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:04 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:04 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:05.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:04 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:15:05.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128227 12.00 MB/sec execute 132 sec latency 49.423 ms 2024-04-27T02:15:06.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:05 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:15:06.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:05 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:15:06.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:05 smithi008 ceph-mon[21873]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:15:06.101 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:05 smithi008 ceph-mon[21873]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:15:06.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:05 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:15:06.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:05 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:15:06.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:05 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:15:06.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:05 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:15:06.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128485 11.96 MB/sec execute 133 sec latency 49.929 ms 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: pgmap v13: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 579 op/s 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.124 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.125 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:06 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: pgmap v13: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 579 op/s 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:06 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:07.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 128905 11.91 MB/sec execute 134 sec latency 66.157 ms 2024-04-27T02:15:08.140 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:07 smithi008 ceph-mon[21873]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-27T02:15:08.140 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:07 smithi008 ceph-mon[21873]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-27T02:15:08.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:07 smithi077 ceph-mon[24318]: Reconfiguring prometheus.smithi008 (dependencies changed)... 2024-04-27T02:15:08.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:07 smithi077 ceph-mon[24318]: Reconfiguring daemon prometheus.smithi008 on smithi008 2024-04-27T02:15:08.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129330 11.88 MB/sec execute 135 sec latency 45.948 ms 2024-04-27T02:15:09.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:08 smithi008 ceph-mon[21873]: pgmap v14: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 771 op/s 2024-04-27T02:15:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:08 smithi077 ceph-mon[24318]: pgmap v14: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 771 op/s 2024-04-27T02:15:09.859 INFO:teuthology.orchestra.run.smithi008.stdout: 5 129749 11.90 MB/sec execute 136 sec latency 60.422 ms 2024-04-27T02:15:10.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130235 11.91 MB/sec execute 137 sec latency 56.034 ms 2024-04-27T02:15:10.976 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:10 smithi008 ceph-mon[21873]: pgmap v15: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 461 op/s 2024-04-27T02:15:11.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:10 smithi077 ceph-mon[24318]: pgmap v15: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 461 op/s 2024-04-27T02:15:11.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130589 11.89 MB/sec execute 138 sec latency 37.846 ms 2024-04-27T02:15:12.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 130829 11.86 MB/sec execute 139 sec latency 40.408 ms 2024-04-27T02:15:13.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:12 smithi008 ceph-mon[21873]: pgmap v16: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-04-27T02:15:13.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:12 smithi077 ceph-mon[24318]: pgmap v16: 97 pgs: 97 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 470 op/s 2024-04-27T02:15:13.510 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:15:13.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131147 11.85 MB/sec execute 140 sec latency 39.414 ms 2024-04-27T02:15:14.417 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:15:14.417 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (116s) 17s ago 10m 17.7M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:15:14.417 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (10m) 17s ago 10m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:15:14.417 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (10m) 23s ago 10m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:15:14.417 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running 17s ago 10m - - 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (4m) 17s ago 4m 39.6M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (4m) 23s ago 4m 13.5M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (52s) 17s ago 12m 480M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283 running (2m) 23s ago 10m 83.6M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 b9fb6e7160ad 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (12m) 17s ago 12m 53.8M 2048M 17.2.0 e1d6a67b021e 454ae324f9a9 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (9m) 23s ago 9m 44.7M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (4m) 17s ago 4m 270M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (4m) 23s ago 4m 83.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (87s) 17s ago 10m 11.2M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (62s) 23s ago 9m 8854k - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 17s ago 9m 453M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 17s ago 8m 385M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:15:14.418 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 17s ago 8m 293M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (7m) 17s ago 7m 415M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (7m) 23s ago 7m 410M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (7m) 23s ago 7m 442M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (6m) 23s ago 6m 398M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (6m) 23s ago 6m 512M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:15:14.419 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (64s) 17s ago 9m 53.5M - 2.33.4 514e6a882f6e 7479c7ec023a 2024-04-27T02:15:14.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131517 11.84 MB/sec execute 141 sec latency 69.546 ms 2024-04-27T02:15:15.049 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:15:15.049 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:15:15.049 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:15:15.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:15:15.050 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:15:15.076 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:14 smithi008 ceph-mon[21873]: pgmap v17: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 695 op/s 2024-04-27T02:15:15.076 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:14 smithi008 ceph-mon[21873]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:15.076 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:14 smithi008 ceph-mon[21873]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:15.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:14 smithi077 ceph-mon[24318]: pgmap v17: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 695 op/s 2024-04-27T02:15:15.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:14 smithi077 ceph-mon[24318]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:15.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:14 smithi077 ceph-mon[24318]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:15.564 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:15:15.564 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:15:15.564 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:15:15.564 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:15:15.565 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:15:15.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 131824 11.83 MB/sec execute 142 sec latency 44.664 ms 2024-04-27T02:15:16.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:15 smithi008 ceph-mon[21873]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:16.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:15 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/1912757055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:16.080 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:15 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:16.186 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:15:16.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:15 smithi077 ceph-mon[24318]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:16.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:15 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1912757055' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:16.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:15 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:16.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132081 11.80 MB/sec execute 143 sec latency 43.032 ms 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: pgmap v18: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 437 op/s 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008:9095"}]: dispatch 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:17.088 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:16 smithi008 ceph-mon[21873]: from='client.? 172.21.15.8:0/219505535' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:15:17.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: pgmap v18: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 437 op/s 2024-04-27T02:15:17.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:17.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008:9095"}]: dispatch 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:17.119 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:16 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/219505535' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:15:17.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132561 11.76 MB/sec execute 144 sec latency 37.136 ms 2024-04-27T02:15:18.105 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:17 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:15:18.106 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:17 smithi008 ceph-mon[21873]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008:9095"}]: dispatch 2024-04-27T02:15:18.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:17 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:15:18.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:17 smithi077 ceph-mon[24318]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi008:9095"}]: dispatch 2024-04-27T02:15:18.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 132886 11.72 MB/sec execute 145 sec latency 38.499 ms 2024-04-27T02:15:19.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:18 smithi008 ceph-mon[21873]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 632 op/s 2024-04-27T02:15:19.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:18 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:15:19.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:18 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:15:19.119 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:18 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:19.189 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:18 smithi077 ceph-mon[24318]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 632 op/s 2024-04-27T02:15:19.189 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:19 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:15:19.189 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:19 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:15:19.189 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:19 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:19.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133211 11.71 MB/sec execute 146 sec latency 53.006 ms 2024-04-27T02:15:20.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:19 smithi008 ceph-mon[21873]: Upgrade: Updating mgr.smithi077.snksrp 2024-04-27T02:15:20.127 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:19 smithi008 ceph-mon[21873]: Deploying daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:15:20.200 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:19 smithi077 ceph-mon[24318]: Upgrade: Updating mgr.smithi077.snksrp 2024-04-27T02:15:20.200 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:19 smithi077 ceph-mon[24318]: Deploying daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:15:20.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133564 11.72 MB/sec execute 147 sec latency 75.003 ms 2024-04-27T02:15:21.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:20 smithi077 ceph-mon[24318]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-27T02:15:21.140 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:20 smithi008 ceph-mon[21873]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-27T02:15:21.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 133928 11.69 MB/sec execute 148 sec latency 63.194 ms 2024-04-27T02:15:22.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134186 11.67 MB/sec execute 149 sec latency 63.652 ms 2024-04-27T02:15:23.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:22 smithi008 ceph-mon[21873]: pgmap v21: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-04-27T02:15:23.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:22 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:23.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:22 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:23.114 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:22 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:23.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:22 smithi077 ceph-mon[24318]: pgmap v21: 97 pgs: 97 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 452 op/s 2024-04-27T02:15:23.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:22 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:23.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:22 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:23.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:22 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:23.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134370 11.63 MB/sec execute 150 sec latency 82.944 ms 2024-04-27T02:15:24.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:24 smithi008 ceph-mon[21873]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-27T02:15:24.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:24 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:24.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:24 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:24.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 134642 11.61 MB/sec execute 151 sec latency 139.456 ms 2024-04-27T02:15:24.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:24 smithi077 ceph-mon[24318]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 649 op/s 2024-04-27T02:15:24.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:24 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:24.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:24 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:25.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135044 11.61 MB/sec execute 152 sec latency 41.690 ms 2024-04-27T02:15:26.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135377 11.61 MB/sec execute 153 sec latency 47.884 ms 2024-04-27T02:15:26.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:26 smithi008 ceph-mon[21873]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 412 op/s 2024-04-27T02:15:27.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:26 smithi077 ceph-mon[24318]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 412 op/s 2024-04-27T02:15:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135630 11.58 MB/sec execute 154 sec latency 39.436 ms 2024-04-27T02:15:28.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 135999 11.53 MB/sec execute 155 sec latency 46.050 ms 2024-04-27T02:15:28.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:28 smithi008 ceph-mon[21873]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-04-27T02:15:28.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:28 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:28.991 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:28 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:29.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:28 smithi077 ceph-mon[24318]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.3 MiB/s wr, 588 op/s 2024-04-27T02:15:29.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:28 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:29.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:28 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:29.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136272 11.48 MB/sec execute 156 sec latency 48.546 ms 2024-04-27T02:15:30.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136493 11.44 MB/sec execute 157 sec latency 67.849 ms 2024-04-27T02:15:31.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:30 smithi008 ceph-mon[21873]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-04-27T02:15:31.039 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:30 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:31.039 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:30 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:31.039 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:30 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:31.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:30 smithi077 ceph-mon[24318]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-04-27T02:15:31.181 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:30 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:31.181 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:30 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:31.181 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:30 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:15:31.863 INFO:teuthology.orchestra.run.smithi008.stdout: 5 136892 11.47 MB/sec execute 158 sec latency 39.001 ms 2024-04-27T02:15:32.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:32.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:32.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 421 op/s 2024-04-27T02:15:32.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:15:32.062 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:15:32.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:15:32.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:15:32.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:15:32.063 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:32 smithi008 ceph-mon[21873]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.2 MiB/s wr, 421 op/s 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:15:32.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:15:32.112 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:32 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/2035590302' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:15:32.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137334 11.47 MB/sec execute 159 sec latency 57.013 ms 2024-04-27T02:15:33.097 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:33 smithi008 ceph-mon[21873]: mgrmap e34: smithi008.exkhsc(active, since 47s), standbys: smithi077.snksrp 2024-04-27T02:15:33.397 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:33 smithi077 ceph-mon[24318]: mgrmap e34: smithi008.exkhsc(active, since 47s), standbys: smithi077.snksrp 2024-04-27T02:15:33.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137693 11.45 MB/sec execute 160 sec latency 35.925 ms 2024-04-27T02:15:34.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 137932 11.43 MB/sec execute 161 sec latency 54.787 ms 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: pgmap v27: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-27T02:15:34.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.exkhsc"}]: dispatch 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.exkhsc"}]': finished 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.snksrp"}]: dispatch 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.snksrp"}]': finished 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T02:15:34.891 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:34 smithi008 ceph-mon[21873]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: pgmap v27: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.exkhsc"}]: dispatch 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi008.exkhsc"}]': finished 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.snksrp"}]: dispatch 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.snksrp"}]': finished 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T02:15:35.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:34 smithi077 ceph-mon[24318]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:15:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 138587 11.49 MB/sec execute 162 sec latency 28.680 ms 2024-04-27T02:15:36.148 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:35 smithi008 ceph-mon[21873]: Upgrade: Setting container_image for all mgr 2024-04-27T02:15:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:35 smithi077 ceph-mon[24318]: Upgrade: Setting container_image for all mgr 2024-04-27T02:15:36.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139050 11.52 MB/sec execute 163 sec latency 44.402 ms 2024-04-27T02:15:37.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:36 smithi008 ceph-mon[21873]: pgmap v29: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-27T02:15:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:36 smithi077 ceph-mon[24318]: pgmap v29: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 437 op/s 2024-04-27T02:15:37.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139500 11.48 MB/sec execute 164 sec latency 49.555 ms 2024-04-27T02:15:38.672 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: Upgrade: Updating mon.smithi008 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: Deploying daemon mon.smithi008 on smithi008 2024-04-27T02:15:38.673 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:38 smithi008 ceph-mon[21873]: pgmap v30: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: Upgrade: Updating mon.smithi008 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: Deploying daemon mon.smithi008 on smithi008 2024-04-27T02:15:38.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:38 smithi077 ceph-mon[24318]: pgmap v30: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T02:15:38.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 139996 11.46 MB/sec execute 165 sec latency 34.589 ms 2024-04-27T02:15:39.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 140631 11.53 MB/sec execute 166 sec latency 45.641 ms 2024-04-27T02:15:40.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141207 11.54 MB/sec execute 167 sec latency 97.778 ms 2024-04-27T02:15:40.934 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:40 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:15:41.185 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:40 smithi008 ceph-mon[21873]: pgmap v31: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T02:15:41.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:40 smithi077 ceph-mon[24318]: pgmap v31: 97 pgs: 97 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T02:15:41.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:41 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[21869]: 2024-04-27T02:15:41.342+0000 7f6dddb13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:15:41.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:41 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[21869]: 2024-04-27T02:15:41.342+0000 7f6dddb13700 -1 mon.smithi008@0(leader) e2 *** Got Signal Terminated *** 2024-04-27T02:15:41.444 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:41 smithi008 podman[63731]: 2024-04-27 02:15:41.414272947 +0000 UTC m=+0.257868576 container died 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, architecture=x86_64, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vendor=Red Hat, Inc., name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:15:41.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 141572 11.54 MB/sec execute 168 sec latency 89.975 ms 2024-04-27T02:15:42.702 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:42 smithi008 podman[63731]: 2024-04-27 02:15:42.537152934 +0000 UTC m=+1.380748572 container cleanup 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:15:42.702 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:42 smithi008 bash[63731]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008 2024-04-27T02:15:42.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142165 11.58 MB/sec execute 169 sec latency 47.162 ms 2024-04-27T02:15:42.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:42 smithi008 podman[63756]: 2024-04-27 02:15:42.876190614 +0000 UTC m=+1.456467805 container remove 454ae324f9a903dc7d12456f4cd1c60b651d8e50b48914603b25ad61dc53314f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, name=centos-stream, io.openshift.expose-services=, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-27T02:15:43.455 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Deactivated successfully. 2024-04-27T02:15:43.455 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:15:43.455 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Consumed 18.179s CPU time. 2024-04-27T02:15:43.713 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 systemd[1]: Starting Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:15:43.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 142612 11.61 MB/sec execute 170 sec latency 43.013 ms 2024-04-27T02:15:43.964 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 podman[63882]: 2024-04-27 02:15:43.80360394 +0000 UTC m=+0.029754356 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:15:43.964 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:43 smithi008 podman[63882]: 2024-04-27 02:15:43.983978018 +0000 UTC m=+0.210128422 container create 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5) 2024-04-27T02:15:44.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 podman[63882]: 2024-04-27 02:15:44.370683483 +0000 UTC m=+0.596833888 container init 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T02:15:44.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 podman[63882]: 2024-04-27 02:15:44.381433736 +0000 UTC m=+0.607584141 container start 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T02:15:44.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T02:15:44.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T02:15:44.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: pidfile_write: ignore empty --pid-file 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: load: jerasure load: lrc 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: RocksDB version: 7.9.2 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Git sha 0 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: DB SUMMARY 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: DB Session ID: HGF9EZR3EBIS0RJPFBCT 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: CURRENT file: CURRENT 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: MANIFEST file: MANIFEST-000015 size: 979 Bytes 2024-04-27T02:15:44.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi008/store.db dir, Total Num: 1, files: 000030.sst 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi008/store.db: 000028.log size: 2260934 ; 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.error_if_exists: 0 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.create_if_missing: 0 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.paranoid_checks: 1 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.env: 0x5646024dcc60 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T02:15:44.489 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.info_log: 0x56460361d960 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.statistics: (nil) 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.use_fsync: 0 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_log_file_size: 0 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_fallocate: 1 2024-04-27T02:15:44.490 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.use_direct_reads: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.db_log_dir: 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.wal_dir: 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T02:15:44.491 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.write_buffer_manager: 0x5646035f01e0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.unordered_write: 0 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T02:15:44.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.row_cache: None 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.wal_filter: None 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.two_write_queues: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.wal_compression: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.atomic_flush: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.log_readahead_size: 0 2024-04-27T02:15:44.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_background_jobs: 2 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_background_compactions: -1 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_subcompactions: 1 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T02:15:44.494 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_open_files: -1 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_background_flushes: -1 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Compression algorithms supported: 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kZSTD supported: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kXpressCompression supported: 0 2024-04-27T02:15:44.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kZlibCompression supported: 1 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kSnappyCompression supported: 1 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kLZ4Compression supported: 1 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: kBZip2Compression supported: 0 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.merge_operator: 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_filter: None 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T02:15:44.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56460361da60) 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_type: 0 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_index_type: 0 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_shortening: 1 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: checksum: 4 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: no_block_cache: 0 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache: 0x564603621090 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_options: 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: capacity : 536870912 2024-04-27T02:15:44.497 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_shard_bits : 4 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: strict_capacity_limit : 0 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_cache_compressed: (nil) 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: persistent_cache: (nil) 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size: 4096 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_size_deviation: 10 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_restart_interval: 16 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: index_block_restart_interval: 1 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_block_size: 4096 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: partition_filters: 0 2024-04-27T02:15:44.498 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: use_delta_encoding: 1 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: filter_policy: bloomfilter 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: whole_key_filtering: 1 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: verify_compression: 0 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: format_version: 5 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_index_compression: 1 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: block_align: 0 2024-04-27T02:15:44.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_auto_readahead_size: 262144 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: prepopulate_block_cache: 0 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: initial_auto_readahead_size: 8192 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression: NoCompression 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.num_levels: 7 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T02:15:44.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:15:44.501 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T02:15:44.502 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T02:15:44.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.table_properties_collectors: 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.inplace_update_support: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.bloom_locality: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.max_successive_merges: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T02:15:44.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.ttl: 2592000 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enable_blob_files: false 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.min_blob_size: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T02:15:44.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi008/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11849, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c1fe1a8e-8e48-45f9-8afd-9e2e357c9aad 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184144424234, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-27T02:15:44.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184144433878, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2062145, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11850, "largest_seqno": 12969, "table_properties": {"data_size": 2056671, "index_size": 3243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 11265, "raw_average_key_size": 23, "raw_value_size": 2046127, "raw_average_value_size": 4253, "num_data_blocks": 149, "num_entries": 481, "num_filter_entries": 481, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714184144, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c1fe1a8e-8e48-45f9-8afd-9e2e357c9aad", "db_session_id": "HGF9EZR3EBIS0RJPFBCT", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184144434044, "job": 1, "event": "recovery_finished"} 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi008/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564603712000 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: DB pointer 0x5646036fc000 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** DB Stats ** 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T02:15:44.507 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L0 1/0 1.97 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: L6 1/0 6.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Sum 2/0 8.94 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** Compaction Stats [default] ** 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 414.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T02:15:44.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Cumulative compaction: 0.00 GB write, 149.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Interval compaction: 0.00 GB write, 149.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache BinnedLRUCache@0x564603621090#2 capacity: 512.00 MB usage: 4.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-27T02:15:44.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: starting mon.smithi008 rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi008 fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???) e2 preinit fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).mds e7 new map 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).mds e7 print_map 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: e7 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: legacy client fscid: 1 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Filesystem 'foofs' (1) 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: fs_name foofs 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: epoch 5 2024-04-27T02:15:44.510 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: created 2024-04-27T02:10:19.132552+0000 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: modified 2024-04-27T02:10:25.155858+0000 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: tableserver 0 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: root 0 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_timeout 60 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: session_autoclose 300 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_file_size 1099511627776 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_xattr_size 65536 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: required_client_features {} 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure 0 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: last_failure_osd_epoch 0 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: max_mds 1 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: in 0 2024-04-27T02:15:44.511 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: up {0=14486} 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: failed 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: damaged 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: stopped 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: data_pools [3] 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: metadata_pool 2 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: inline_data disabled 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: balancer 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: bal_rank_mask -1 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: standby_count_wanted 1 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: qdb_cluster leader: 0 members: 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi008.wzyiku{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: Standby daemons: 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: 2024-04-27T02:15:44.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout: [mds.foofs.smithi077.qcnfuu{-1:24293} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:15:44.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-27T02:15:44.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:15:44.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:15:44.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:15:44.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 ceph-mon[63900]: mon.smithi008@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-04-27T02:15:44.590 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 bash[63882]: 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 2024-04-27T02:15:44.590 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:44 smithi008 systemd[1]: Started Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:15:44.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143052 11.57 MB/sec execute 171 sec latency 75.636 ms 2024-04-27T02:15:45.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143502 11.54 MB/sec execute 172 sec latency 42.156 ms 2024-04-27T02:15:46.644 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:15:46.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143556 11.49 MB/sec execute 173 sec latency 871.648 ms 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (2m) 16s ago 11m 17.7M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 16s ago 11m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (10m) 18s ago 10m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 unknown 16s ago 11m - - 2024-04-27T02:15:47.352 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (5m) 16s ago 5m 39.7M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (5m) 18s ago 5m 13.7M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (85s) 16s ago 12m 496M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (25s) 18s ago 10m 111M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 starting - - - 2048M 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (10m) 18s ago 10m 42.8M 2048M 17.2.0 e1d6a67b021e 8fd9749efc76 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (5m) 16s ago 5m 278M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:15:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (5m) 18s ago 5m 87.6M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 16s ago 10m 12.8M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (94s) 18s ago 10m 14.7M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (9m) 16s ago 9m 492M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 16s ago 9m 405M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (8m) 16s ago 8m 305M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (8m) 16s ago 8m 444M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:15:47.354 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (8m) 18s ago 8m 481M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:15:47.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (7m) 18s ago 7m 508M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:15:47.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (7m) 18s ago 7m 424M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:15:47.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (6m) 18s ago 6m 323M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:15:47.355 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (32s) 16s ago 10m 49.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:15:47.790 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:15:47.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:15:47.790 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:15:47.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:15:47.791 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:15:47.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143556 11.42 MB/sec execute 174 sec latency 1871.702 ms 2024-04-27T02:15:48.133 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:15:48.133 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:15:48.134 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:15:48.583 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:15:48.583 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:15:48.583 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-27T02:15:48.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143556 11.36 MB/sec execute 175 sec latency 2871.756 ms 2024-04-27T02:15:49.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143556 11.29 MB/sec execute 176 sec latency 3871.785 ms 2024-04-27T02:15:50.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 143658 11.25 MB/sec execute 177 sec latency 4657.695 ms 2024-04-27T02:15:50.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 847 op/s 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.6 MiB/s wr, 645 op/s 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: pgmap v34: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1801768545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 828 op/s 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1656200904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:15:50.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: pgmap v36: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: mon.smithi008 calling monitor election 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: mon.smithi008 is new leader, mons smithi008,smithi077 in quorum (ranks 0,1) 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: monmap epoch 2 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: last_changed 2024-04-27T02:05:17.114745+0000 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: created 2024-04-27T02:02:59.719455+0000 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: min_mon_release 17 (quincy) 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: election_strategy: 1 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:15:50.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: mgrmap e34: smithi008.exkhsc(active, since 66s), standbys: smithi077.snksrp 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='mgr.14758 ' entity='' 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: mgrmap e35: smithi008.exkhsc(active, since 66s), standbys: smithi077.snksrp 2024-04-27T02:15:50.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:50 smithi008 ceph-mon[63900]: from='mgr.14758 ' entity='' 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='mgr.14758 172.21.15.8:0/3097457233' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 847 op/s 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.6 MiB/s wr, 645 op/s 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: pgmap v34: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-04-27T02:15:50.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.24603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1801768545' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 828 op/s 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='client.? 172.21.15.8:0/1656200904' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: pgmap v36: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: mon.smithi008 calling monitor election 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: mon.smithi008 is new leader, mons smithi008,smithi077 in quorum (ranks 0,1) 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: monmap epoch 2 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: last_changed 2024-04-27T02:05:17.114745+0000 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: created 2024-04-27T02:02:59.719455+0000 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: min_mon_release 17 (quincy) 2024-04-27T02:15:50.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: election_strategy: 1 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: mgrmap e34: smithi008.exkhsc(active, since 66s), standbys: smithi077.snksrp 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: daemon grafana.smithi008 on smithi008 is in unknown state 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='mgr.14758 ' entity='' 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: mgrmap e35: smithi008.exkhsc(active, since 66s), standbys: smithi077.snksrp 2024-04-27T02:15:50.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:50 smithi077 ceph-mon[24318]: from='mgr.14758 ' entity='' 2024-04-27T02:15:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144186 11.29 MB/sec execute 178 sec latency 38.733 ms 2024-04-27T02:15:52.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:52 smithi008 ceph-mon[63900]: mgrmap e36: smithi008.exkhsc(active, since 67s), standbys: smithi077.snksrp 2024-04-27T02:15:52.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 144635 11.29 MB/sec execute 179 sec latency 45.503 ms 2024-04-27T02:15:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:52 smithi077 ceph-mon[24318]: mgrmap e36: smithi008.exkhsc(active, since 67s), standbys: smithi077.snksrp 2024-04-27T02:15:53.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145041 11.29 MB/sec execute 180 sec latency 26.698 ms 2024-04-27T02:15:54.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145578 11.32 MB/sec execute 181 sec latency 43.638 ms 2024-04-27T02:15:55.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 145875 11.31 MB/sec execute 182 sec latency 50.199 ms 2024-04-27T02:15:56.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146113 11.29 MB/sec execute 183 sec latency 61.665 ms 2024-04-27T02:15:57.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146360 11.26 MB/sec execute 184 sec latency 60.626 ms 2024-04-27T02:15:58.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 146656 11.21 MB/sec execute 185 sec latency 53.098 ms 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:15:59.191 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:15:59 smithi008 ceph-mon[63900]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp restarted 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: Standby manager daemon smithi077.snksrp started 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/crt"}]: dispatch 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.snksrp/key"}]: dispatch 2024-04-27T02:15:59.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:15:59 smithi077 ceph-mon[24318]: from='mgr.? 172.21.15.77:0/3834909827' entity='mgr.smithi077.snksrp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T02:15:59.864 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147013 11.19 MB/sec execute 186 sec latency 42.898 ms 2024-04-27T02:16:00.204 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:00 smithi008 ceph-mon[63900]: mgrmap e37: smithi008.exkhsc(active, since 74s), standbys: smithi077.snksrp 2024-04-27T02:16:00.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:00 smithi077 ceph-mon[24318]: mgrmap e37: smithi008.exkhsc(active, since 74s), standbys: smithi077.snksrp 2024-04-27T02:16:00.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147454 11.21 MB/sec execute 187 sec latency 33.106 ms 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: Active manager daemon smithi008.exkhsc restarted 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: mgrmap e38: smithi008.exkhsc(active, starting, since 0.0163385s), standbys: smithi077.snksrp 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:16:01.213 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:16:01.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: Active manager daemon smithi008.exkhsc restarted 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: Activating manager daemon smithi008.exkhsc 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: mgrmap e38: smithi008.exkhsc(active, starting, since 0.0163385s), standbys: smithi077.snksrp 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi008.exkhsc", "id": "smithi008.exkhsc"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr metadata", "who": "smithi077.snksrp", "id": "smithi077.snksrp"}]: dispatch 2024-04-27T02:16:01.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T02:16:01.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:01 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T02:16:01.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 147849 11.21 MB/sec execute 188 sec latency 57.643 ms 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:16:02.220 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:02 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: Manager daemon smithi008.exkhsc is now available 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/mirror_snapshot_schedule"}]: dispatch 2024-04-27T02:16:02.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:16:02.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:02 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi008.exkhsc/trash_purge_schedule"}]: dispatch 2024-04-27T02:16:02.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148316 11.21 MB/sec execute 189 sec latency 34.131 ms 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: [27/Apr/2024:02:16:01] ENGINE Bus STARTING 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: [27/Apr/2024:02:16:02] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: mgrmap e39: smithi008.exkhsc(active, since 1.07869s), standbys: smithi077.snksrp 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: [27/Apr/2024:02:16:02] ENGINE Serving on http://172.21.15.8:8765 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: [27/Apr/2024:02:16:02] ENGINE Bus STARTED 2024-04-27T02:16:03.229 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:03 smithi008 ceph-mon[63900]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:03.382 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: [27/Apr/2024:02:16:01] ENGINE Bus STARTING 2024-04-27T02:16:03.382 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: [27/Apr/2024:02:16:02] ENGINE Serving on https://172.21.15.8:7150 2024-04-27T02:16:03.382 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: mgrmap e39: smithi008.exkhsc(active, since 1.07869s), standbys: smithi077.snksrp 2024-04-27T02:16:03.382 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: [27/Apr/2024:02:16:02] ENGINE Serving on http://172.21.15.8:8765 2024-04-27T02:16:03.382 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: [27/Apr/2024:02:16:02] ENGINE Bus STARTED 2024-04-27T02:16:03.383 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:03 smithi077 ceph-mon[24318]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:03.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 148648 11.21 MB/sec execute 190 sec latency 25.720 ms 2024-04-27T02:16:04.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:04 smithi008 ceph-mon[63900]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:04.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:04 smithi008 ceph-mon[63900]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:16:04.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:04 smithi008 ceph-mon[63900]: Cluster is now healthy 2024-04-27T02:16:04.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:04 smithi008 ceph-mon[63900]: mgrmap e40: smithi008.exkhsc(active, since 2s), standbys: smithi077.snksrp 2024-04-27T02:16:04.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:04 smithi077 ceph-mon[24318]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:04.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:04 smithi077 ceph-mon[24318]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:16:04.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:04 smithi077 ceph-mon[24318]: Cluster is now healthy 2024-04-27T02:16:04.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:04 smithi077 ceph-mon[24318]: mgrmap e40: smithi008.exkhsc(active, since 2s), standbys: smithi077.snksrp 2024-04-27T02:16:04.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149096 11.21 MB/sec execute 191 sec latency 38.258 ms 2024-04-27T02:16:05.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149393 11.21 MB/sec execute 192 sec latency 77.928 ms 2024-04-27T02:16:06.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:06 smithi008 ceph-mon[63900]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:06.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:06 smithi008 ceph-mon[63900]: mgrmap e41: smithi008.exkhsc(active, since 4s), standbys: smithi077.snksrp 2024-04-27T02:16:06.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:06 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:06.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:06 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:06.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:06 smithi077 ceph-mon[24318]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:06.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:06 smithi077 ceph-mon[24318]: mgrmap e41: smithi008.exkhsc(active, since 4s), standbys: smithi077.snksrp 2024-04-27T02:16:06.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:06 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:06.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:06 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:06.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149619 11.19 MB/sec execute 193 sec latency 44.703 ms 2024-04-27T02:16:07.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 149872 11.18 MB/sec execute 194 sec latency 63.171 ms 2024-04-27T02:16:08.549 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:08 smithi008 ceph-mon[63900]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:08.550 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:08 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:08.550 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:08 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:08.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:08 smithi077 ceph-mon[24318]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-27T02:16:08.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:08 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:08.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:08 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:08.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150197 11.13 MB/sec execute 195 sec latency 57.300 ms 2024-04-27T02:16:09.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150447 11.09 MB/sec execute 196 sec latency 76.854 ms 2024-04-27T02:16:10.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:10 smithi008 ceph-mon[63900]: pgmap v7: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.2 MiB/s wr, 386 op/s 2024-04-27T02:16:10.636 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:10 smithi077 ceph-mon[24318]: pgmap v7: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.2 MiB/s wr, 386 op/s 2024-04-27T02:16:10.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150647 11.06 MB/sec execute 197 sec latency 47.105 ms 2024-04-27T02:16:11.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:11 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:11 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:11 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.647 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:11 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 150964 11.07 MB/sec execute 198 sec latency 37.625 ms 2024-04-27T02:16:11.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:11 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:11 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:11 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:11.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:11 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:12.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:12 smithi008 ceph-mon[63900]: pgmap v8: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 299 op/s 2024-04-27T02:16:12.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:12 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:12.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:12 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:12.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:12.659 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:16:12.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151463 11.09 MB/sec execute 199 sec latency 41.838 ms 2024-04-27T02:16:12.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:12 smithi077 ceph-mon[24318]: pgmap v8: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 299 op/s 2024-04-27T02:16:12.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:12 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:12.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:12 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:12.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:12 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:12.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:12 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:16:13.663 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.664 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:13 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:13.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 151943 11.08 MB/sec execute 200 sec latency 50.425 ms 2024-04-27T02:16:13.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.conf 2024-04-27T02:16:14.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.conf 2024-04-27T02:16:14.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:16:14.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T02:16:14.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:16:14.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:13 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:14.644 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:16:14.644 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:14.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:14 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T02:16:14.677 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: Updating smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: Updating smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:14.678 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T02:16:14.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152463 11.11 MB/sec execute 201 sec latency 25.148 ms 2024-04-27T02:16:15.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 152974 11.14 MB/sec execute 202 sec latency 30.803 ms 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: Upgrade: Updating mon.smithi077 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: Deploying daemon mon.smithi077 on smithi077 2024-04-27T02:16:16.703 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:16.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153404 11.15 MB/sec execute 203 sec latency 36.868 ms 2024-04-27T02:16:16.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.6 MiB/s wr, 460 op/s 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: Upgrade: Updating mon.smithi077 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: from='mgr.34104 ' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: Deploying daemon mon.smithi077 on smithi077 2024-04-27T02:16:16.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:16 smithi077 ceph-mon[24318]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:17.800 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:17 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:16:17.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 153869 11.12 MB/sec execute 204 sec latency 38.341 ms 2024-04-27T02:16:18.141 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077[24313]: 2024-04-27T02:16:17.821+0000 7fdfc16e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:16:18.142 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077[24313]: 2024-04-27T02:16:17.821+0000 7fdfc16e7700 -1 mon.smithi077@1(peon) e2 *** Got Signal Terminated *** 2024-04-27T02:16:18.142 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:17 smithi077 podman[50162]: 2024-04-27 02:16:17.872519365 +0000 UTC m=+0.168741069 container died 8fd9749efc7635b56fc400b2e8435bddc02c9e34c375852e3477c555227bda0b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, distribution-scope=public, vcs-type=git, version=8) 2024-04-27T02:16:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 podman[50162]: 2024-04-27 02:16:18.262850017 +0000 UTC m=+0.559071712 container cleanup 8fd9749efc7635b56fc400b2e8435bddc02c9e34c375852e3477c555227bda0b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, release=754, io.buildah.version=1.19.8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, io.openshift.expose-services=, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container) 2024-04-27T02:16:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 bash[50162]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077 2024-04-27T02:16:18.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 154145 11.09 MB/sec execute 205 sec latency 36.260 ms 2024-04-27T02:16:18.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 podman[50175]: 2024-04-27 02:16:18.547967435 +0000 UTC m=+0.670651097 container remove 8fd9749efc7635b56fc400b2e8435bddc02c9e34c375852e3477c555227bda0b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, name=centos-stream, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vendor=Red Hat, Inc., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-27T02:16:18.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi077.service: Deactivated successfully. 2024-04-27T02:16:18.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 systemd[1]: Stopped Ceph mon.smithi077 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:16:18.913 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi077.service: Consumed 7.135s CPU time. 2024-04-27T02:16:19.133 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:16:19.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:18 smithi077 systemd[1]: Starting Ceph mon.smithi077 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:16:19.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 podman[50306]: 2024-04-27 02:16:19.060908934 +0000 UTC m=+0.018772289 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:16:19.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 podman[50306]: 2024-04-27 02:16:19.182726137 +0000 UTC m=+0.140589471 container create 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T02:16:19.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 podman[50306]: 2024-04-27 02:16:19.419441706 +0000 UTC m=+0.377305049 container init 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T02:16:19.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 podman[50306]: 2024-04-27 02:16:19.422611308 +0000 UTC m=+0.380474647 container start 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb) 2024-04-27T02:16:19.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T02:16:19.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T02:16:19.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: pidfile_write: ignore empty --pid-file 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: load: jerasure load: lrc 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: RocksDB version: 7.9.2 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Git sha 0 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: DB SUMMARY 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: DB Session ID: OVPF715083TH2YKRB8KB 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: CURRENT file: CURRENT 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: MANIFEST file: MANIFEST-000009 size: 904 Bytes 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 1, files: 000024.sst 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 000022.log size: 6623176 ; 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.error_if_exists: 0 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.create_if_missing: 0 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.paranoid_checks: 1 2024-04-27T02:16:19.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.env: 0x55e79230bc60 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.info_log: 0x55e794121960 2024-04-27T02:16:19.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.statistics: (nil) 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.use_fsync: 0 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_log_file_size: 0 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_fallocate: 1 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T02:16:19.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.use_direct_reads: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.db_log_dir: 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.wal_dir: 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T02:16:19.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.write_buffer_manager: 0x55e7940f41e0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.unordered_write: 0 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.row_cache: None 2024-04-27T02:16:19.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.wal_filter: None 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.two_write_queues: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.wal_compression: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.atomic_flush: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.log_readahead_size: 0 2024-04-27T02:16:19.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_background_jobs: 2 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_background_compactions: -1 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_subcompactions: 1 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T02:16:19.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_open_files: -1 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_background_flushes: -1 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Compression algorithms supported: 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kZSTD supported: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kXpressCompression supported: 0 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kZlibCompression supported: 1 2024-04-27T02:16:19.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kSnappyCompression supported: 1 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kLZ4Compression supported: 1 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: kBZip2Compression supported: 0 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000009 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.merge_operator: 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_filter: None 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T02:16:19.743 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e794121a60) 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_type: 0 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: data_block_index_type: 0 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_shortening: 1 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: checksum: 4 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: no_block_cache: 0 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache: 0x55e794125090 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_options: 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: capacity : 536870912 2024-04-27T02:16:19.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: num_shard_bits : 4 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: strict_capacity_limit : 0 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_compressed: (nil) 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: persistent_cache: (nil) 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_size: 4096 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_size_deviation: 10 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_restart_interval: 16 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_block_restart_interval: 1 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: metadata_block_size: 4096 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: partition_filters: 0 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: use_delta_encoding: 1 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: filter_policy: bloomfilter 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: whole_key_filtering: 1 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: verify_compression: 0 2024-04-27T02:16:19.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: format_version: 5 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: enable_index_compression: 1 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_align: 0 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: max_auto_readahead_size: 262144 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: prepopulate_block_cache: 0 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: initial_auto_readahead_size: 8192 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression: NoCompression 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.num_levels: 7 2024-04-27T02:16:19.746 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T02:16:19.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T02:16:19.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T02:16:19.749 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T02:16:19.749 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T02:16:19.749 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T02:16:19.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.table_properties_collectors: 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.inplace_update_support: 0 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.bloom_locality: 0 2024-04-27T02:16:19.751 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.max_successive_merges: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.ttl: 2592000 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enable_blob_files: false 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.min_blob_size: 0 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T02:16:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12219, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d482ff53-3bf8-4993-9e2c-9610ef439b05 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184179455521, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184179474374, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4009547, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12224, "largest_seqno": 14013, "table_properties": {"data_size": 4001820, "index_size": 4915, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1797, "raw_key_size": 16749, "raw_average_key_size": 23, "raw_value_size": 3986381, "raw_average_value_size": 5575, "num_data_blocks": 223, "num_entries": 715, "num_filter_entries": 715, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714184179, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d482ff53-3bf8-4993-9e2c-9610ef439b05", "db_session_id": "OVPF715083TH2YKRB8KB", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714184179474494, "job": 1, "event": "recovery_finished"} 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-04-27T02:16:19.753 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi077/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e794216000 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: DB pointer 0x55e794200000 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** DB Stats ** 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T02:16:19.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: L0 1/0 3.82 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 479.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: L6 1/0 6.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Sum 2/0 10.80 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 479.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 479.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 479.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T02:16:19.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative compaction: 0.00 GB write, 170.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval compaction: 0.00 GB write, 170.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Block cache BinnedLRUCache@0x55e794125090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T02:16:19.756 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: starting mon.smithi077 rank 1 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi077 fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???) e2 preinit fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).mds e7 new map 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).mds e7 print_map 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: e7 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: legacy client fscid: 1 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Filesystem 'foofs' (1) 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: fs_name foofs 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: epoch 5 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: created 2024-04-27T02:10:19.132552+0000 2024-04-27T02:16:19.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: modified 2024-04-27T02:10:25.155858+0000 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: tableserver 0 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: root 0 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: session_timeout 60 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: session_autoclose 300 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: max_file_size 1099511627776 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: max_xattr_size 65536 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: required_client_features {} 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: last_failure 0 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: last_failure_osd_epoch 0 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: max_mds 1 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: in 0 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: up {0=14486} 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: failed 2024-04-27T02:16:19.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: damaged 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: stopped 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: data_pools [3] 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: metadata_pool 2 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: inline_data disabled 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: balancer 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: bal_rank_mask -1 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: standby_count_wanted 1 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: qdb_cluster leader: 0 members: 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: [mds.foofs.smithi008.wzyiku{0:14486} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.8:6834/4094915007,v1:172.21.15.8:6835/4094915007] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Standby daemons: 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: [mds.foofs.smithi077.qcnfuu{-1:24293} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T02:16:19.759 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 ceph-mon[50325]: mon.smithi077@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 bash[50306]: 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 2024-04-27T02:16:19.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:19 smithi077 systemd[1]: Started Ceph mon.smithi077 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:16:19.865 INFO:teuthology.orchestra.run.smithi008.stdout: 5 154498 11.11 MB/sec execute 206 sec latency 45.724 ms 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 9s ago 11m 17.3M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (11m) 9s ago 11m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (11m) 14s ago 11m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 9s ago 11m - - 2024-04-27T02:16:20.235 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (5m) 9s ago 5m 40.7M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (5m) 14s ago 5m 14.0M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (117s) 9s ago 13m 478M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (58s) 14s ago 11m 447M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (36s) 9s ago 13m 35.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 starting - - - 2048M 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (5m) 9s ago 5m 277M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (5m) 14s ago 5m 87.6M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 9s ago 11m 13.5M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (2m) 14s ago 10m 17.3M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 9s ago 10m 524M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (9m) 9s ago 9m 452M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (9m) 9s ago 9m 348M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (8m) 9s ago 8m 492M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (8m) 14s ago 8m 496M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:16:20.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (8m) 14s ago 8m 555M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:16:20.237 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (7m) 14s ago 7m 488M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:16:20.237 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (7m) 14s ago 7m 350M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:16:20.237 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (65s) 9s ago 10m 57.2M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:16:20.250 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:20 smithi008 ceph-mon[63900]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 31 KiB/s rd, 8.1 MiB/s wr, 793 op/s 2024-04-27T02:16:20.250 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:20.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:20.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:20 smithi077 ceph-mon[50325]: pgmap v12: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 31 KiB/s rd, 8.1 MiB/s wr, 793 op/s 2024-04-27T02:16:20.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:20.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:20.869 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155153 11.14 MB/sec execute 207 sec latency 38.982 ms 2024-04-27T02:16:20.900 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:16:20.900 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:16:20.900 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:16:20.901 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:16:21.289 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: mgrmap e41: smithi008.exkhsc(active, since 18s), standbys: smithi077.snksrp 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:21.290 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: mon.smithi008 calling monitor election 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: mon.smithi077 calling monitor election 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: mon.smithi008 is new leader, mons smithi008,smithi077 in quorum (ranks 0,1) 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: monmap epoch 3 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:16:21.291 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: last_changed 2024-04-27T02:16:20.058904+0000 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: created 2024-04-27T02:02:59.719455+0000 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: min_mon_release 19 (squid) 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: election_strategy: 1 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:21.292 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: mgrmap e41: smithi008.exkhsc(active, since 19s), standbys: smithi077.snksrp 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:21.293 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:21.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:21.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:21 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2396793021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:21.398 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "3/23 daemons upgraded", 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:16:21.399 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: mgrmap e41: smithi008.exkhsc(active, since 18s), standbys: smithi077.snksrp 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: mon.smithi008 calling monitor election 2024-04-27T02:16:21.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: mon.smithi077 calling monitor election 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: mon.smithi008 is new leader, mons smithi008,smithi077 in quorum (ranks 0,1) 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: monmap epoch 3 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: fsid 27e8cf32-043a-11ef-bc93-c7b262605968 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: last_changed 2024-04-27T02:16:20.058904+0000 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: created 2024-04-27T02:02:59.719455+0000 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: min_mon_release 19 (squid) 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: election_strategy: 1 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: 0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.smithi008 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: mgrmap e41: smithi008.exkhsc(active, since 19s), standbys: smithi077.snksrp 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:21.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi008"}]: dispatch 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:21.486 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:21 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2396793021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:21.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155508 11.11 MB/sec execute 208 sec latency 83.454 ms 2024-04-27T02:16:21.970 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:21.970 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:21.970 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:22.294 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:22 smithi008 ceph-mon[63900]: from='client.34142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:22.295 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:22 smithi008 ceph-mon[63900]: pgmap v14: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 538 op/s 2024-04-27T02:16:22.295 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:22 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/879983148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:16:22.374 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:22 smithi077 ceph-mon[50325]: from='client.34142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:22.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:22 smithi077 ceph-mon[50325]: pgmap v14: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 538 op/s 2024-04-27T02:16:22.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:22 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/879983148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:16:22.869 INFO:teuthology.orchestra.run.smithi008.stdout: 5 155858 11.10 MB/sec execute 209 sec latency 36.845 ms 2024-04-27T02:16:23.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156224 11.12 MB/sec execute 210 sec latency 39.941 ms 2024-04-27T02:16:24.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:24 smithi008 ceph-mon[63900]: pgmap v15: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 625 op/s 2024-04-27T02:16:24.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:24 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:24.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156483 11.11 MB/sec execute 211 sec latency 80.729 ms 2024-04-27T02:16:24.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:24 smithi077 ceph-mon[50325]: pgmap v15: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 625 op/s 2024-04-27T02:16:24.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:24 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:25.776 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:25.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 156799 11.12 MB/sec execute 212 sec latency 40.904 ms 2024-04-27T02:16:25.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:26.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:26 smithi008 ceph-mon[63900]: pgmap v16: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 543 op/s 2024-04-27T02:16:26.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:26 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:26.783 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:26 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:26.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157223 11.08 MB/sec execute 213 sec latency 35.402 ms 2024-04-27T02:16:26.879 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:26 smithi077 ceph-mon[50325]: pgmap v16: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 543 op/s 2024-04-27T02:16:26.879 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:26 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:26.879 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:26 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 157612 11.06 MB/sec execute 214 sec latency 36.812 ms 2024-04-27T02:16:28.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:28 smithi008 ceph-mon[63900]: pgmap v17: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 753 op/s 2024-04-27T02:16:28.869 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158123 11.09 MB/sec execute 215 sec latency 42.276 ms 2024-04-27T02:16:28.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:28 smithi077 ceph-mon[50325]: pgmap v17: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 753 op/s 2024-04-27T02:16:29.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 158721 11.12 MB/sec execute 216 sec latency 40.126 ms 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: pgmap v18: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 463 op/s 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:30.576 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:30.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: pgmap v18: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 463 op/s 2024-04-27T02:16:30.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:30.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:30.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:30.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159162 11.10 MB/sec execute 217 sec latency 28.089 ms 2024-04-27T02:16:31.581 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:31 smithi008 ceph-mon[63900]: pgmap v19: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-27T02:16:31.581 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:31 smithi008 ceph-mon[63900]: pgmap v20: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.6 MiB/s wr, 662 op/s 2024-04-27T02:16:31.581 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:31 smithi008 ceph-mon[63900]: Reconfiguring mon.smithi008 (monmap changed)... 2024-04-27T02:16:31.581 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:31 smithi008 ceph-mon[63900]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-27T02:16:31.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:31 smithi077 ceph-mon[50325]: pgmap v19: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-27T02:16:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:31 smithi077 ceph-mon[50325]: pgmap v20: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.6 MiB/s wr, 662 op/s 2024-04-27T02:16:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:31 smithi077 ceph-mon[50325]: Reconfiguring mon.smithi008 (monmap changed)... 2024-04-27T02:16:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:31 smithi077 ceph-mon[50325]: Reconfiguring daemon mon.smithi008 on smithi008 2024-04-27T02:16:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:31.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 159653 11.13 MB/sec execute 218 sec latency 45.133 ms 2024-04-27T02:16:32.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160010 11.13 MB/sec execute 219 sec latency 65.371 ms 2024-04-27T02:16:33.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160405 11.15 MB/sec execute 220 sec latency 35.451 ms 2024-04-27T02:16:33.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.1 MiB/s wr, 299 op/s 2024-04-27T02:16:33.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:33.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:33.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:16:33.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:16:33.903 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.1 MiB/s wr, 299 op/s 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi008.exkhsc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:16:34.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:34.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 160774 11.11 MB/sec execute 221 sec latency 54.033 ms 2024-04-27T02:16:34.915 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:34 smithi008 ceph-mon[63900]: Reconfiguring mgr.smithi008.exkhsc (monmap changed)... 2024-04-27T02:16:34.915 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:34 smithi008 ceph-mon[63900]: Reconfiguring daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:16:35.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:34 smithi077 ceph-mon[50325]: Reconfiguring mgr.smithi008.exkhsc (monmap changed)... 2024-04-27T02:16:35.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:34 smithi077 ceph-mon[50325]: Reconfiguring daemon mgr.smithi008.exkhsc on smithi008 2024-04-27T02:16:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161038 11.08 MB/sec execute 222 sec latency 76.810 ms 2024-04-27T02:16:35.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:35 smithi008 ceph-mon[63900]: pgmap v22: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 722 op/s 2024-04-27T02:16:35.927 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:35.928 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:35.928 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:16:35.928 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:35 smithi077 ceph-mon[50325]: pgmap v22: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 722 op/s 2024-04-27T02:16:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:36.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:16:36.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:36.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161330 11.07 MB/sec execute 223 sec latency 54.184 ms 2024-04-27T02:16:36.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:36 smithi008 ceph-mon[63900]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-27T02:16:36.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:36 smithi008 ceph-mon[63900]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-27T02:16:36.954 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:36 smithi008 ceph-mon[63900]: pgmap v23: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 422 op/s 2024-04-27T02:16:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:36 smithi077 ceph-mon[50325]: Reconfiguring crash.smithi008 (monmap changed)... 2024-04-27T02:16:37.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:36 smithi077 ceph-mon[50325]: Reconfiguring daemon crash.smithi008 on smithi008 2024-04-27T02:16:37.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:36 smithi077 ceph-mon[50325]: pgmap v23: 97 pgs: 97 active+clean; 351 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 422 op/s 2024-04-27T02:16:37.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 161649 11.08 MB/sec execute 224 sec latency 42.278 ms 2024-04-27T02:16:38.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162067 11.09 MB/sec execute 225 sec latency 57.498 ms 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: pgmap v24: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 731 op/s 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: Reconfiguring osd.0 (monmap changed)... 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:39.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:39 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.0 on smithi008 2024-04-27T02:16:39.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: pgmap v24: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 731 op/s 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: Reconfiguring osd.0 (monmap changed)... 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:39.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:39 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.0 on smithi008 2024-04-27T02:16:39.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162339 11.06 MB/sec execute 226 sec latency 78.105 ms 2024-04-27T02:16:40.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 162727 11.05 MB/sec execute 227 sec latency 39.932 ms 2024-04-27T02:16:41.563 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: pgmap v25: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.2 MiB/s wr, 614 op/s 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: Reconfiguring osd.1 (monmap changed)... 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:41.564 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:41 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.1 on smithi008 2024-04-27T02:16:41.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: pgmap v25: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.2 MiB/s wr, 614 op/s 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: Reconfiguring osd.1 (monmap changed)... 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:41.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:41 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.1 on smithi008 2024-04-27T02:16:41.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163173 11.06 MB/sec execute 228 sec latency 56.016 ms 2024-04-27T02:16:42.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 163607 11.07 MB/sec execute 229 sec latency 28.736 ms 2024-04-27T02:16:43.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:43 smithi008 ceph-mon[63900]: pgmap v26: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.1 MiB/s wr, 511 op/s 2024-04-27T02:16:43.628 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:43 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:43.628 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:43 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:43.628 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:43 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:16:43.628 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:43 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:43 smithi077 ceph-mon[50325]: pgmap v26: 97 pgs: 97 active+clean; 347 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.1 MiB/s wr, 511 op/s 2024-04-27T02:16:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:43 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:43 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:43 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:16:43.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:43 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:43.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164089 11.09 MB/sec execute 230 sec latency 42.032 ms 2024-04-27T02:16:44.652 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:44 smithi008 ceph-mon[63900]: Reconfiguring osd.2 (monmap changed)... 2024-04-27T02:16:44.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:44 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.2 on smithi008 2024-04-27T02:16:44.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:44 smithi077 ceph-mon[50325]: Reconfiguring osd.2 (monmap changed)... 2024-04-27T02:16:44.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:44 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.2 on smithi008 2024-04-27T02:16:44.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164552 11.06 MB/sec execute 231 sec latency 32.754 ms 2024-04-27T02:16:45.679 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:45 smithi008 ceph-mon[63900]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 731 op/s 2024-04-27T02:16:45.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:45 smithi077 ceph-mon[50325]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 731 op/s 2024-04-27T02:16:45.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 164935 11.07 MB/sec execute 232 sec latency 30.770 ms 2024-04-27T02:16:46.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165354 11.06 MB/sec execute 233 sec latency 36.398 ms 2024-04-27T02:16:47.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 435 op/s 2024-04-27T02:16:47.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: Reconfiguring osd.3 (monmap changed)... 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.3 on smithi008 2024-04-27T02:16:47.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:47.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 435 op/s 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: Reconfiguring osd.3 (monmap changed)... 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.3 on smithi008 2024-04-27T02:16:47.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:16:47.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 165909 11.09 MB/sec execute 234 sec latency 29.607 ms 2024-04-27T02:16:48.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166268 11.08 MB/sec execute 235 sec latency 46.967 ms 2024-04-27T02:16:49.252 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:49 smithi008 ceph-mon[63900]: pgmap v29: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 719 op/s 2024-04-27T02:16:49.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:49.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:49.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:16:49.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:49 smithi077 ceph-mon[50325]: pgmap v29: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 719 op/s 2024-04-27T02:16:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:16:49.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:49.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166490 11.05 MB/sec execute 236 sec latency 51.334 ms 2024-04-27T02:16:50.164 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:50 smithi008 ceph-mon[63900]: Reconfiguring mds.foofs.smithi008.wzyiku (monmap changed)... 2024-04-27T02:16:50.165 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:50 smithi008 ceph-mon[63900]: Reconfiguring daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:16:50.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:50 smithi077 ceph-mon[50325]: Reconfiguring mds.foofs.smithi008.wzyiku (monmap changed)... 2024-04-27T02:16:50.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:50 smithi077 ceph-mon[50325]: Reconfiguring daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:16:50.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 166819 11.06 MB/sec execute 237 sec latency 46.533 ms 2024-04-27T02:16:51.339 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:51 smithi008 ceph-mon[63900]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-04-27T02:16:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:51 smithi077 ceph-mon[50325]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-04-27T02:16:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167104 11.04 MB/sec execute 238 sec latency 51.269 ms 2024-04-27T02:16:52.535 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:16:52.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:52.605 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:52.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: Reconfiguring crash.smithi077 (monmap changed)... 2024-04-27T02:16:52.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:16:52.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:52.606 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:52 smithi008 ceph-mon[63900]: Reconfiguring daemon crash.smithi077 on smithi077 2024-04-27T02:16:52.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:52.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:52.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: Reconfiguring crash.smithi077 (monmap changed)... 2024-04-27T02:16:52.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:16:52.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:52.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:52 smithi077 ceph-mon[50325]: Reconfiguring daemon crash.smithi077 on smithi077 2024-04-27T02:16:52.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167423 11.05 MB/sec execute 239 sec latency 62.585 ms 2024-04-27T02:16:53.611 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: pgmap v31: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:16:53.612 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:53 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:53.615 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:16:53.615 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (3m) 43s ago 12m 17.3M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:16:53.615 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (12m) 43s ago 12m 6400k - 17.2.0 e1d6a67b021e 19f0ac89895a 2024-04-27T02:16:53.615 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (11m) 29s ago 11m 6405k - 17.2.0 e1d6a67b021e f4c4dedd6b5f 2024-04-27T02:16:53.615 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 43s ago 12m - - 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (6m) 43s ago 6m 40.7M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (6m) 29s ago 6m 14.1M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (2m) 43s ago 13m 478M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (91s) 29s ago 11m 448M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (69s) 43s ago 13m 35.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (34s) 29s ago 11m 23.5M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (6m) 43s ago 6m 277M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (6m) 29s ago 6m 88.1M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (3m) 43s ago 12m 13.5M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (2m) 29s ago 11m 20.5M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (10m) 43s ago 10m 524M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10m) 43s ago 10m 452M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:16:53.616 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (9m) 43s ago 9m 348M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (9m) 43s ago 9m 492M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (9m) 29s ago 9m 512M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (8m) 29s ago 8m 310M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (8m) 29s ago 8m 505M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (7m) 29s ago 7m 373M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:16:53.617 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (98s) 43s ago 11m 57.2M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:16:53.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: pgmap v31: 97 pgs: 97 active+clean; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-04-27T02:16:53.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.snksrp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T02:16:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T02:16:53.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:53 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:53.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167706 11.03 MB/sec execute 240 sec latency 83.639 ms 2024-04-27T02:16:54.256 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:16:54.257 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:16:54.258 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T02:16:54.258 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T02:16:54.258 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:16:54.258 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:16:54.492 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:54 smithi008 ceph-mon[63900]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:54.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:54 smithi008 ceph-mon[63900]: Reconfiguring mgr.smithi077.snksrp (monmap changed)... 2024-04-27T02:16:54.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:54 smithi008 ceph-mon[63900]: Reconfiguring daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:16:54.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:54 smithi008 ceph-mon[63900]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:54.493 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:54 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3179589306' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:54 smithi077 ceph-mon[50325]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:54 smithi077 ceph-mon[50325]: Reconfiguring mgr.smithi077.snksrp (monmap changed)... 2024-04-27T02:16:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:54 smithi077 ceph-mon[50325]: Reconfiguring daemon mgr.smithi077.snksrp on smithi077 2024-04-27T02:16:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:54 smithi077 ceph-mon[50325]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:54 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3179589306' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:16:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:16:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:16:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "4/23 daemons upgraded", 2024-04-27T02:16:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mon daemons", 2024-04-27T02:16:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:16:54.818 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:16:54.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 167958 10.99 MB/sec execute 241 sec latency 46.518 ms 2024-04-27T02:16:55.400 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:16:55.400 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:16:55.400 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:16:55.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: pgmap v32: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 705 op/s 2024-04-27T02:16:55.508 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:55.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:55.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:55.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:55.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:55.509 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:55 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/4017181545' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:16:55.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: pgmap v32: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 705 op/s 2024-04-27T02:16:55.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T02:16:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T02:16:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:55.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:55 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/4017181545' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:16:55.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168203 10.96 MB/sec execute 242 sec latency 57.207 ms 2024-04-27T02:16:56.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:56 smithi077 ceph-mon[50325]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:56.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:56 smithi077 ceph-mon[50325]: Reconfiguring mon.smithi077 (monmap changed)... 2024-04-27T02:16:56.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:56 smithi077 ceph-mon[50325]: Reconfiguring daemon mon.smithi077 on smithi077 2024-04-27T02:16:56.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:56 smithi008 ceph-mon[63900]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:16:56.537 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:56 smithi008 ceph-mon[63900]: Reconfiguring mon.smithi077 (monmap changed)... 2024-04-27T02:16:56.538 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:56 smithi008 ceph-mon[63900]: Reconfiguring daemon mon.smithi077 on smithi077 2024-04-27T02:16:56.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168502 10.96 MB/sec execute 243 sec latency 68.389 ms 2024-04-27T02:16:57.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:57 smithi008 ceph-mon[63900]: pgmap v33: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-04-27T02:16:57.599 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:57.600 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:57.600 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:16:57.600 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:57.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:57 smithi077 ceph-mon[50325]: pgmap v33: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-04-27T02:16:57.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:57.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:57.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:16:57.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:57.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 168791 10.95 MB/sec execute 244 sec latency 68.522 ms 2024-04-27T02:16:58.609 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:58 smithi008 ceph-mon[63900]: Reconfiguring osd.4 (monmap changed)... 2024-04-27T02:16:58.609 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:58 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.4 on smithi077 2024-04-27T02:16:58.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:58 smithi077 ceph-mon[50325]: Reconfiguring osd.4 (monmap changed)... 2024-04-27T02:16:58.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:58 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.4 on smithi077 2024-04-27T02:16:58.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169134 10.95 MB/sec execute 245 sec latency 42.826 ms 2024-04-27T02:16:59.627 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:59 smithi008 ceph-mon[63900]: pgmap v34: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 660 op/s 2024-04-27T02:16:59.629 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:59 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:59.629 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:59 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:59.629 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:59 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:16:59.629 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:16:59 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:59.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:59 smithi077 ceph-mon[50325]: pgmap v34: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 660 op/s 2024-04-27T02:16:59.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:59 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:59.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:59 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:16:59.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:59 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:16:59.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:16:59 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:16:59.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169425 10.94 MB/sec execute 246 sec latency 43.268 ms 2024-04-27T02:17:00.632 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:00 smithi077 ceph-mon[50325]: Reconfiguring osd.5 (monmap changed)... 2024-04-27T02:17:00.632 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:00 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.5 on smithi077 2024-04-27T02:17:00.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:00 smithi008 ceph-mon[63900]: Reconfiguring osd.5 (monmap changed)... 2024-04-27T02:17:00.653 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:00 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.5 on smithi077 2024-04-27T02:17:00.870 INFO:teuthology.orchestra.run.smithi008.stdout: 5 169776 10.92 MB/sec execute 247 sec latency 48.520 ms 2024-04-27T02:17:01.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170012 10.90 MB/sec execute 248 sec latency 42.298 ms 2024-04-27T02:17:01.937 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: pgmap v35: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 376 op/s 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: Reconfiguring osd.6 (monmap changed)... 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.6 on smithi077 2024-04-27T02:17:01.938 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: pgmap v35: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 376 op/s 2024-04-27T02:17:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: Reconfiguring osd.6 (monmap changed)... 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.6 on smithi077 2024-04-27T02:17:01.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:02.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170334 10.91 MB/sec execute 249 sec latency 44.149 ms 2024-04-27T02:17:03.526 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: pgmap v36: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 376 op/s 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: Reconfiguring osd.7 (monmap changed)... 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:03.527 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:03 smithi008 ceph-mon[63900]: Reconfiguring daemon osd.7 on smithi077 2024-04-27T02:17:03.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: pgmap v36: 97 pgs: 97 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 376 op/s 2024-04-27T02:17:03.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:03.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:03.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: Reconfiguring osd.7 (monmap changed)... 2024-04-27T02:17:03.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:17:03.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:03.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:03 smithi077 ceph-mon[50325]: Reconfiguring daemon osd.7 on smithi077 2024-04-27T02:17:03.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170563 10.89 MB/sec execute 250 sec latency 53.233 ms 2024-04-27T02:17:04.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 170815 10.87 MB/sec execute 251 sec latency 59.297 ms 2024-04-27T02:17:05.347 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: pgmap v37: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-04-27T02:17:05.347 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:05.347 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:05.348 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: Reconfiguring mds.foofs.smithi077.qcnfuu (monmap changed)... 2024-04-27T02:17:05.348 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:17:05.348 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:05.348 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:05 smithi008 ceph-mon[63900]: Reconfiguring daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:17:05.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: pgmap v37: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-04-27T02:17:05.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:05.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:05.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: Reconfiguring mds.foofs.smithi077.qcnfuu (monmap changed)... 2024-04-27T02:17:05.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:17:05.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:05.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:05 smithi077 ceph-mon[50325]: Reconfiguring daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:17:05.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171114 10.87 MB/sec execute 252 sec latency 74.259 ms 2024-04-27T02:17:06.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171328 10.84 MB/sec execute 253 sec latency 117.167 ms 2024-04-27T02:17:07.161 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: pgmap v38: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.4 MiB/s wr, 356 op/s 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all mon 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi077"}]: dispatch 2024-04-27T02:17:07.162 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi077"}]': finished 2024-04-27T02:17:07.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: pgmap v38: 97 pgs: 97 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.4 MiB/s wr, 356 op/s 2024-04-27T02:17:07.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all mon 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]: dispatch 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi008"}]': finished 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi077"}]: dispatch 2024-04-27T02:17:07.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi077"}]': finished 2024-04-27T02:17:07.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171626 10.82 MB/sec execute 254 sec latency 35.941 ms 2024-04-27T02:17:08.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 171956 10.81 MB/sec execute 255 sec latency 49.671 ms 2024-04-27T02:17:09.693 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: pgmap v39: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-04-27T02:17:09.693 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: Upgrade: Updating crash.smithi008 (1/2) 2024-04-27T02:17:09.694 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:09.694 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:17:09.694 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:09.694 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:09 smithi008 ceph-mon[63900]: Deploying daemon crash.smithi008 on smithi008 2024-04-27T02:17:09.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172282 10.81 MB/sec execute 256 sec latency 43.298 ms 2024-04-27T02:17:09.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: pgmap v39: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-04-27T02:17:09.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: Upgrade: Updating crash.smithi008 (1/2) 2024-04-27T02:17:09.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:09.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi008", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:17:09.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:09.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:09 smithi077 ceph-mon[50325]: Deploying daemon crash.smithi008 on smithi008 2024-04-27T02:17:10.871 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172681 10.81 MB/sec execute 257 sec latency 39.787 ms 2024-04-27T02:17:11.732 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:11 smithi008 ceph-mon[63900]: pgmap v40: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 343 op/s 2024-04-27T02:17:11.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 172941 10.80 MB/sec execute 258 sec latency 53.889 ms 2024-04-27T02:17:11.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:11 smithi077 ceph-mon[50325]: pgmap v40: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 343 op/s 2024-04-27T02:17:12.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173246 10.78 MB/sec execute 259 sec latency 51.739 ms 2024-04-27T02:17:13.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:13 smithi008 ceph-mon[63900]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 343 op/s 2024-04-27T02:17:13.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173500 10.76 MB/sec execute 260 sec latency 56.955 ms 2024-04-27T02:17:13.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:13 smithi077 ceph-mon[50325]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.5 MiB/s wr, 343 op/s 2024-04-27T02:17:14.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 173724 10.73 MB/sec execute 261 sec latency 87.425 ms 2024-04-27T02:17:15.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:15 smithi008 ceph-mon[63900]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:17:15.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:15 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:15.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:15 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:15.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174060 10.75 MB/sec execute 262 sec latency 38.886 ms 2024-04-27T02:17:15.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:15 smithi077 ceph-mon[50325]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-04-27T02:17:15.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:15 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:15.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:15 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:16.845 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:16.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174410 10.75 MB/sec execute 263 sec latency 58.569 ms 2024-04-27T02:17:16.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:17.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 174751 10.75 MB/sec execute 264 sec latency 56.951 ms 2024-04-27T02:17:17.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:17 smithi008 ceph-mon[63900]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.6 MiB/s wr, 365 op/s 2024-04-27T02:17:17.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:17.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:17:17.890 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:17.969 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:17 smithi077 ceph-mon[50325]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.6 MiB/s wr, 365 op/s 2024-04-27T02:17:17.970 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:17.970 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T02:17:17.970 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:18.829 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:18 smithi077 ceph-mon[50325]: Upgrade: Updating crash.smithi077 (2/2) 2024-04-27T02:17:18.829 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:18 smithi077 ceph-mon[50325]: Deploying daemon crash.smithi077 on smithi077 2024-04-27T02:17:18.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175128 10.73 MB/sec execute 265 sec latency 53.528 ms 2024-04-27T02:17:18.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:18 smithi008 ceph-mon[63900]: Upgrade: Updating crash.smithi077 (2/2) 2024-04-27T02:17:18.904 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:18 smithi008 ceph-mon[63900]: Deploying daemon crash.smithi077 on smithi077 2024-04-27T02:17:19.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 175637 10.74 MB/sec execute 266 sec latency 35.811 ms 2024-04-27T02:17:19.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:19 smithi008 ceph-mon[63900]: pgmap v44: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 546 op/s 2024-04-27T02:17:19.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:19 smithi077 ceph-mon[50325]: pgmap v44: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 546 op/s 2024-04-27T02:17:20.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176105 10.75 MB/sec execute 267 sec latency 45.178 ms 2024-04-27T02:17:21.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176520 10.75 MB/sec execute 268 sec latency 36.315 ms 2024-04-27T02:17:21.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:21 smithi008 ceph-mon[63900]: pgmap v45: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-27T02:17:21.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:21.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:21.922 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:21 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:21.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:21 smithi077 ceph-mon[50325]: pgmap v45: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-27T02:17:21.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:21.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:21.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:21 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:22.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 176963 10.75 MB/sec execute 269 sec latency 29.967 ms 2024-04-27T02:17:23.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177339 10.75 MB/sec execute 270 sec latency 40.501 ms 2024-04-27T02:17:23.953 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:23 smithi008 ceph-mon[63900]: pgmap v46: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-27T02:17:23.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:23 smithi077 ceph-mon[50325]: pgmap v46: 97 pgs: 97 active+clean; 347 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 385 op/s 2024-04-27T02:17:24.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 177787 10.77 MB/sec execute 271 sec latency 44.633 ms 2024-04-27T02:17:25.737 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:25 smithi008 ceph-mon[63900]: pgmap v47: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 645 op/s 2024-04-27T02:17:25.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:25.738 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:25.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178188 10.77 MB/sec execute 272 sec latency 40.981 ms 2024-04-27T02:17:25.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:25 smithi077 ceph-mon[50325]: pgmap v47: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 645 op/s 2024-04-27T02:17:25.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:25.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:25.991 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:17:26.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178516 10.75 MB/sec execute 273 sec latency 49.868 ms 2024-04-27T02:17:27.080 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 76s ago 13m 17.3M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 starting - - - - 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (6s) 1s ago 12m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 76s ago 12m - - 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (7m) 76s ago 7m 40.7M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (7m) 1s ago 7m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (3m) 76s ago 14m 478M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (2m) 1s ago 12m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (103s) 76s ago 14m 35.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:17:27.081 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (68s) 1s ago 12m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (6m) 76s ago 6m 277M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (6m) 1s ago 6m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (3m) 76s ago 12m 13.5M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (3m) 1s ago 12m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (11m) 76s ago 11m 524M 1937M 17.2.0 e1d6a67b021e 73b5228008ce 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (10m) 76s ago 10m 452M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (10m) 76s ago 10m 348M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 76s ago 10m 492M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (9m) 1s ago 9m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (9m) 1s ago 9m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (8m) 1s ago 8m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (8m) 1s ago 8m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:17:27.082 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 76s ago 11m 57.2M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:17:27.771 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:27 smithi008 ceph-mon[63900]: pgmap v48: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-04-27T02:17:27.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:27 smithi008 ceph-mon[63900]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:27.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:27 smithi008 ceph-mon[63900]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:27.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:27.772 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:17:27.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:17:27.773 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:17:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 178814 10.74 MB/sec execute 274 sec latency 39.426 ms 2024-04-27T02:17:27.910 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:27 smithi077 ceph-mon[50325]: pgmap v48: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-04-27T02:17:27.910 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:27 smithi077 ceph-mon[50325]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:27.910 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:27 smithi077 ceph-mon[50325]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:27.910 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:27.910 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:28.353 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:17:28.353 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:17:28.353 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:17:28.353 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "5/23 daemons upgraded", 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading crash daemons", 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:17:28.354 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:17:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:28 smithi008 ceph-mon[63900]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:28.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:28 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/356790276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:28.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179139 10.73 MB/sec execute 275 sec latency 78.414 ms 2024-04-27T02:17:28.939 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:28 smithi077 ceph-mon[50325]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:28.939 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:28 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/356790276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:29.091 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:17:29.091 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:17:29.091 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:17:29.824 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:29 smithi008 ceph-mon[63900]: pgmap v49: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.9 MiB/s wr, 681 op/s 2024-04-27T02:17:29.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:29 smithi008 ceph-mon[63900]: from='client.44131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:29.825 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:29 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/4162382723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:17:29.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179504 10.73 MB/sec execute 276 sec latency 46.363 ms 2024-04-27T02:17:29.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:29 smithi077 ceph-mon[50325]: pgmap v49: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.9 MiB/s wr, 681 op/s 2024-04-27T02:17:29.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:29 smithi077 ceph-mon[50325]: from='client.44131 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:17:29.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:29 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/4162382723' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:17:30.872 INFO:teuthology.orchestra.run.smithi008.stdout: 5 179899 10.74 MB/sec execute 277 sec latency 40.522 ms 2024-04-27T02:17:31.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: pgmap v50: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T02:17:31.355 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:31.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.356 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: pgmap v50: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:31.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180208 10.72 MB/sec execute 278 sec latency 48.634 ms 2024-04-27T02:17:32.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 180684 10.72 MB/sec execute 279 sec latency 38.700 ms 2024-04-27T02:17:33.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:32 smithi008 ceph-mon[63900]: pgmap v51: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T02:17:33.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:33.113 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:32 smithi077 ceph-mon[50325]: pgmap v51: 97 pgs: 97 active+clean; 351 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T02:17:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:33.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181282 10.76 MB/sec execute 280 sec latency 30.463 ms 2024-04-27T02:17:34.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 181934 10.78 MB/sec execute 281 sec latency 29.662 ms 2024-04-27T02:17:35.131 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:34 smithi008 ceph-mon[63900]: pgmap v52: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-27T02:17:35.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:34 smithi008 ceph-mon[63900]: mgrmap e42: smithi008.exkhsc(active, since 92s), standbys: smithi077.snksrp 2024-04-27T02:17:35.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:34 smithi077 ceph-mon[50325]: pgmap v52: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-27T02:17:35.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:34 smithi077 ceph-mon[50325]: mgrmap e42: smithi008.exkhsc(active, since 92s), standbys: smithi077.snksrp 2024-04-27T02:17:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182584 10.79 MB/sec execute 282 sec latency 33.292 ms 2024-04-27T02:17:36.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 182861 10.78 MB/sec execute 283 sec latency 72.416 ms 2024-04-27T02:17:37.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:36 smithi008 ceph-mon[63900]: pgmap v53: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-27T02:17:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:36 smithi077 ceph-mon[50325]: pgmap v53: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 452 op/s 2024-04-27T02:17:37.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183166 10.77 MB/sec execute 284 sec latency 60.590 ms 2024-04-27T02:17:38.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183449 10.77 MB/sec execute 285 sec latency 52.505 ms 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: pgmap v54: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 775 op/s 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: pgmap v55: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 634 op/s 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: pgmap v56: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 790 op/s 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.195 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all crash 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi077"}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi077"}]': finished 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T02:17:39.196 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:39 smithi008 ceph-mon[63900]: Upgrade: osd.0 is safe to restart 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: pgmap v54: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 775 op/s 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: pgmap v55: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 634 op/s 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: pgmap v56: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 790 op/s 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:39.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all crash 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi008"}]': finished 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi077"}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi077"}]': finished 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T02:17:39.485 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:39 smithi077 ceph-mon[50325]: Upgrade: osd.0 is safe to restart 2024-04-27T02:17:39.873 INFO:teuthology.orchestra.run.smithi008.stdout: 5 183739 10.75 MB/sec execute 286 sec latency 66.455 ms 2024-04-27T02:17:40.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184378 10.77 MB/sec execute 287 sec latency 33.857 ms 2024-04-27T02:17:41.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: pgmap v57: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 790 op/s 2024-04-27T02:17:41.710 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: Upgrade: Updating osd.0 2024-04-27T02:17:41.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:41.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:17:41.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:41.711 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:41 smithi008 ceph-mon[63900]: Deploying daemon osd.0 on smithi008 2024-04-27T02:17:41.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 184998 10.80 MB/sec execute 288 sec latency 28.638 ms 2024-04-27T02:17:41.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: pgmap v57: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 790 op/s 2024-04-27T02:17:41.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: Upgrade: Updating osd.0 2024-04-27T02:17:41.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:41.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T02:17:41.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:17:41.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:41 smithi077 ceph-mon[50325]: Deploying daemon osd.0 on smithi008 2024-04-27T02:17:42.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185364 10.81 MB/sec execute 289 sec latency 63.976 ms 2024-04-27T02:17:43.743 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:43 smithi008 ceph-mon[63900]: pgmap v58: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-04-27T02:17:43.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185594 10.78 MB/sec execute 290 sec latency 57.379 ms 2024-04-27T02:17:43.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:43 smithi077 ceph-mon[50325]: pgmap v58: 97 pgs: 97 active+clean; 355 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 477 op/s 2024-04-27T02:17:44.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 185877 10.77 MB/sec execute 291 sec latency 52.051 ms 2024-04-27T02:17:45.802 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:45 smithi008 ceph-mon[63900]: pgmap v59: 97 pgs: 97 active+clean; 356 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 856 op/s 2024-04-27T02:17:45.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186300 10.76 MB/sec execute 292 sec latency 39.587 ms 2024-04-27T02:17:45.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:45 smithi077 ceph-mon[50325]: pgmap v59: 97 pgs: 97 active+clean; 356 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 856 op/s 2024-04-27T02:17:46.807 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:46.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 186808 10.77 MB/sec execute 293 sec latency 40.120 ms 2024-04-27T02:17:46.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:17:47.314 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:47 smithi008 systemd[1]: Stopping Ceph osd.0 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:17:47.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:47 smithi008 ceph-mon[63900]: pgmap v60: 97 pgs: 97 active+clean; 356 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 378 op/s 2024-04-27T02:17:47.816 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:47 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[32348]: 2024-04-27T02:17:47.768+0000 7efc355a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:17:47.816 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:47 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[32348]: 2024-04-27T02:17:47.768+0000 7efc355a1700 -1 osd.0 73 *** Got signal Terminated *** 2024-04-27T02:17:47.817 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:47 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[32348]: 2024-04-27T02:17:47.768+0000 7efc355a1700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:17:47.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187303 10.78 MB/sec execute 294 sec latency 29.410 ms 2024-04-27T02:17:47.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:47 smithi077 ceph-mon[50325]: pgmap v60: 97 pgs: 97 active+clean; 356 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 378 op/s 2024-04-27T02:17:48.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:48 smithi008 ceph-mon[63900]: osd.0 marked itself down and dead 2024-04-27T02:17:48.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187793 10.78 MB/sec execute 295 sec latency 169.782 ms 2024-04-27T02:17:48.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:48 smithi077 ceph-mon[50325]: osd.0 marked itself down and dead 2024-04-27T02:17:49.625 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:49 smithi008 podman[69497]: 2024-04-27 02:17:49.393948888 +0000 UTC m=+1.921550624 container died 73b5228008ce07698ebdc178c0952fefda32e4615766baddacf535ef1fa56696 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, version=8, io.openshift.expose-services=, release=754, vcs-type=git) 2024-04-27T02:17:49.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 187842 10.75 MB/sec execute 296 sec latency 1054.402 ms 2024-04-27T02:17:49.883 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:49 smithi008 ceph-mon[63900]: pgmap v61: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 589 op/s 2024-04-27T02:17:49.883 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:49 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:17:49.883 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:49 smithi008 ceph-mon[63900]: osdmap e74: 8 total, 7 up, 8 in 2024-04-27T02:17:49.883 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:49 smithi008 podman[69497]: 2024-04-27 02:17:49.747353914 +0000 UTC m=+2.274955653 container cleanup 73b5228008ce07698ebdc178c0952fefda32e4615766baddacf535ef1fa56696 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., ceph=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-04-27T02:17:49.883 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:49 smithi008 bash[69497]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0 2024-04-27T02:17:50.204 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:50 smithi008 podman[69515]: 2024-04-27 02:17:50.175379727 +0000 UTC m=+0.781464485 container remove 73b5228008ce07698ebdc178c0952fefda32e4615766baddacf535ef1fa56696 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, RELEASE=HEAD, name=centos-stream) 2024-04-27T02:17:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:49 smithi077 ceph-mon[50325]: pgmap v61: 97 pgs: 97 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 589 op/s 2024-04-27T02:17:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:49 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:17:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:49 smithi077 ceph-mon[50325]: osdmap e74: 8 total, 7 up, 8 in 2024-04-27T02:17:50.643 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:50 smithi008 podman[69604]: 2024-04-27 02:17:50.463170116 +0000 UTC m=+0.023470427 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:17:50.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188308 10.78 MB/sec execute 297 sec latency 41.347 ms 2024-04-27T02:17:50.894 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:50 smithi008 ceph-mon[63900]: osdmap e75: 8 total, 7 up, 8 in 2024-04-27T02:17:50.894 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:50 smithi008 podman[69604]: 2024-04-27 02:17:50.777746769 +0000 UTC m=+0.338047072 container create 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T02:17:51.148 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:51 smithi008 podman[69604]: 2024-04-27 02:17:51.149406261 +0000 UTC m=+0.709706566 container init 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-27T02:17:51.148 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:51 smithi008 podman[69604]: 2024-04-27 02:17:51.162527947 +0000 UTC m=+0.722828251 container start 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T02:17:51.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:50 smithi077 ceph-mon[50325]: osdmap e75: 8 total, 7 up, 8 in 2024-04-27T02:17:51.399 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:51 smithi008 podman[69604]: 2024-04-27 02:17:51.333527845 +0000 UTC m=+0.893828149 container attach 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T02:17:51.650 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:51 smithi008 podman[69604]: 2024-04-27 02:17:51.661942356 +0000 UTC m=+1.222242662 container died 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T02:17:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 188836 10.79 MB/sec execute 298 sec latency 38.145 ms 2024-04-27T02:17:51.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:51 smithi008 ceph-mon[63900]: pgmap v64: 97 pgs: 13 stale+active+clean, 84 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 736 op/s 2024-04-27T02:17:52.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:51 smithi077 ceph-mon[50325]: pgmap v64: 97 pgs: 13 stale+active+clean, 84 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 736 op/s 2024-04-27T02:17:52.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189273 10.78 MB/sec execute 299 sec latency 32.963 ms 2024-04-27T02:17:52.916 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:52 smithi008 podman[69629]: 2024-04-27 02:17:52.710841292 +0000 UTC m=+1.033751814 container remove 2da1fad2d2b40bee484c6700cb14dfc48f454925c7a582290ba289d956848600 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:17:52.916 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:52 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-27T02:17:52.916 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:52 smithi008 systemd[1]: Stopped Ceph osd.0 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:17:52.917 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:52 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.0.service: Consumed 3min 26.374s CPU time. 2024-04-27T02:17:53.175 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:53 smithi008 systemd[1]: Starting Ceph osd.0 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:17:53.429 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:53 smithi008 podman[69722]: 2024-04-27 02:17:53.275100207 +0000 UTC m=+0.023177100 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:17:53.429 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:53 smithi008 podman[69722]: 2024-04-27 02:17:53.3871866 +0000 UTC m=+0.135263482 container create 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T02:17:53.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189548 10.77 MB/sec execute 300 sec latency 46.807 ms 2024-04-27T02:17:53.957 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:53 smithi008 ceph-mon[63900]: pgmap v65: 97 pgs: 13 stale+active+clean, 84 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 352 op/s 2024-04-27T02:17:53.957 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:53 smithi008 podman[69722]: 2024-04-27 02:17:53.908895932 +0000 UTC m=+0.656972808 container init 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-27T02:17:53.957 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:53 smithi008 podman[69722]: 2024-04-27 02:17:53.919842396 +0000 UTC m=+0.667919276 container start 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T02:17:54.212 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 podman[69722]: 2024-04-27 02:17:54.070641689 +0000 UTC m=+0.818718571 container attach 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-27T02:17:54.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:53 smithi077 ceph-mon[50325]: pgmap v65: 97 pgs: 13 stale+active+clean, 84 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 352 op/s 2024-04-27T02:17:54.479 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 bash[69722]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T02:17:54.480 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate[69737]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-27T02:17:54.730 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:54 smithi008 podman[69722]: 2024-04-27 02:17:54.513089335 +0000 UTC m=+1.261166209 container died 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb) 2024-04-27T02:17:54.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 189776 10.76 MB/sec execute 301 sec latency 84.040 ms 2024-04-27T02:17:54.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-mon[63900]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.0 MiB/s wr, 712 op/s; 220/1677 objects degraded (13.119%) 2024-04-27T02:17:54.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:54 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 220/1677 objects degraded (13.119%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T02:17:55.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:54 smithi077 ceph-mon[50325]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.0 MiB/s wr, 712 op/s; 220/1677 objects degraded (13.119%) 2024-04-27T02:17:55.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:54 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 220/1677 objects degraded (13.119%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T02:17:55.784 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:55 smithi008 podman[69905]: 2024-04-27 02:17:55.62979815 +0000 UTC m=+1.105529538 container remove 7c96167757a5adce6edea565c74f539bbfca24338514204f1f455f9b63405efc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T02:17:55.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190062 10.75 MB/sec execute 302 sec latency 67.803 ms 2024-04-27T02:17:56.036 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:55 smithi008 podman[69967]: 2024-04-27 02:17:55.788034149 +0000 UTC m=+0.026017131 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:17:56.036 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:55 smithi008 podman[69967]: 2024-04-27 02:17:55.972229392 +0000 UTC m=+0.210212357 container create f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T02:17:56.286 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 podman[69967]: 2024-04-27 02:17:56.285217182 +0000 UTC m=+0.523200160 container init f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-27T02:17:56.286 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 podman[69967]: 2024-04-27 02:17:56.291119383 +0000 UTC m=+0.529102355 container start f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb) 2024-04-27T02:17:56.552 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 ceph-osd[69985]: -- 172.21.15.8:0/2577997812 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f99ed5f680 con 0x55f99ed29000 2024-04-27T02:17:56.552 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 bash[69967]: f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c 2024-04-27T02:17:56.552 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 systemd[1]: Started Ceph osd.0 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:17:56.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190324 10.74 MB/sec execute 303 sec latency 47.946 ms 2024-04-27T02:17:57.065 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:17:56 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:17:56.860+0000 7ff7f2c7f7c0 -1 Falling back to public interface 2024-04-27T02:17:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:57 smithi008 ceph-mon[63900]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 360 op/s; 220/1677 objects degraded (13.119%) 2024-04-27T02:17:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:57.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:57.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190614 10.74 MB/sec execute 304 sec latency 60.683 ms 2024-04-27T02:17:57.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:57 smithi077 ceph-mon[50325]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 351 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 360 op/s; 220/1677 objects degraded (13.119%) 2024-04-27T02:17:57.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:57.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:17:57.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:17:58.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 190961 10.72 MB/sec execute 305 sec latency 70.903 ms 2024-04-27T02:17:59.672 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:17:59.704 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:17:59 smithi008 ceph-mon[63900]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 551 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:17:59.874 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191232 10.70 MB/sec execute 306 sec latency 51.640 ms 2024-04-27T02:17:59.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:17:59 smithi077 ceph-mon[50325]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 551 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:18:00.728 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:00 smithi008 ceph-mon[63900]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:00.847 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (4m) 30s ago 13m 17.3M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (46s) 30s ago 13m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (40s) 35s ago 12m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 30s ago 13m - - 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (7m) 30s ago 7m 40.4M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (7m) 35s ago 7m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (3m) 30s ago 14m 500M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (2m) 35s ago 12m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 30s ago 14m 45.1M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (101s) 35s ago 12m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (7m) 30s ago 7m 272M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (7m) 35s ago 7m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 30s ago 13m 13.2M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:18:00.848 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (3m) 35s ago 12m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 starting - - - 1937M 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (11m) 30s ago 11m 547M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 30s ago 11m 436M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (10m) 30s ago 10m 546M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (10m) 35s ago 10m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (9m) 35s ago 9m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (9m) 35s ago 9m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (9m) 35s ago 9m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:18:00.849 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 30s ago 12m 60.4M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:18:00.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191411 10.68 MB/sec execute 307 sec latency 69.000 ms 2024-04-27T02:18:00.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:00 smithi077 ceph-mon[50325]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:01.529 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:18:01.530 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:18:01.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:01 smithi008 ceph-mon[63900]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 499 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:18:01.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:01 smithi008 ceph-mon[63900]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:01.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:01.740 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:01.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 191696 10.68 MB/sec execute 308 sec latency 48.043 ms 2024-04-27T02:18:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:01 smithi077 ceph-mon[50325]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 499 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:18:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:01 smithi077 ceph-mon[50325]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:02.137 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:18:02.137 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "6/23 daemons upgraded", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:18:02.138 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:18:02.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:02 smithi008 ceph-mon[63900]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:02.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:02 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/394124352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:02.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192038 10.68 MB/sec execute 309 sec latency 54.865 ms 2024-04-27T02:18:02.884 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 204/1542 objects degraded (13.230%), 29 pgs degraded 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: osd.0 (root=default,host=smithi008) is down 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 204/1542 objects degraded (13.230%), 29 pgs degraded 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2024-04-27T02:18:02.885 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-04-27T02:18:02.886 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-04-27T02:18:02.887 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-04-27T02:18:02.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:02 smithi077 ceph-mon[50325]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:02.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:02 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/394124352' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:03.000 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:18:02 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:18:02.825+0000 7ff7f2c7f7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-27T02:18:03.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:03 smithi008 ceph-mon[63900]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 428 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:18:03.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:03 smithi008 ceph-mon[63900]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:03.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:03 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/25341103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:18:03.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192485 10.69 MB/sec execute 310 sec latency 34.524 ms 2024-04-27T02:18:03.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:03 smithi077 ceph-mon[50325]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 428 op/s; 204/1542 objects degraded (13.230%) 2024-04-27T02:18:03.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:03 smithi077 ceph-mon[50325]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:03.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:03 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/25341103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:18:04.530 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:04 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 204/1542 objects degraded (13.230%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:04.530 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:04 smithi008 ceph-mon[63900]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:18:04.531 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:18:04 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:18:04.487+0000 7ff7f2c7f7c0 -1 osd.0 73 log_to_monitors true 2024-04-27T02:18:04.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 192736 10.67 MB/sec execute 311 sec latency 57.710 ms 2024-04-27T02:18:04.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:04 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 204/1542 objects degraded (13.230%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:04.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:04 smithi077 ceph-mon[50325]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T02:18:05.032 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:18:04 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:18:04.877+0000 7ff7e00cf700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:18:05.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:05 smithi008 ceph-mon[63900]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 602 op/s; 216/1620 objects degraded (13.333%) 2024-04-27T02:18:05.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:05 smithi008 ceph-mon[63900]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:18:05.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:05 smithi008 ceph-mon[63900]: osdmap e76: 8 total, 7 up, 8 in 2024-04-27T02:18:05.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:05 smithi008 ceph-mon[63900]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:18:05.536 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:05 smithi008 ceph-mon[63900]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' 2024-04-27T02:18:05.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193034 10.66 MB/sec execute 312 sec latency 49.262 ms 2024-04-27T02:18:05.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:05 smithi077 ceph-mon[50325]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 602 op/s; 216/1620 objects degraded (13.333%) 2024-04-27T02:18:05.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:05 smithi077 ceph-mon[50325]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T02:18:05.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:05 smithi077 ceph-mon[50325]: osdmap e76: 8 total, 7 up, 8 in 2024-04-27T02:18:05.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:05 smithi077 ceph-mon[50325]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:18:05.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:05 smithi077 ceph-mon[50325]: from='osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407]' entity='osd.0' 2024-04-27T02:18:06.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193200 10.64 MB/sec execute 313 sec latency 1008.427 ms 2024-04-27T02:18:06.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:06.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:06.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:18:06.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407] boot 2024-04-27T02:18:06.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: osdmap e77: 8 total, 8 up, 8 in 2024-04-27T02:18:06.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:18:06.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:06.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:06.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:18:06.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: osd.0 [v2:172.21.15.8:6802/187979407,v1:172.21.15.8:6803/187979407] boot 2024-04-27T02:18:06.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: osdmap e77: 8 total, 8 up, 8 in 2024-04-27T02:18:06.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T02:18:07.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:07 smithi008 ceph-mon[63900]: pgmap v74: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.5 MiB/s wr, 261 op/s; 216/1620 objects degraded (13.333%) 2024-04-27T02:18:07.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:07 smithi008 ceph-mon[63900]: osdmap e78: 8 total, 8 up, 8 in 2024-04-27T02:18:07.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:07.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:07 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:07.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193408 10.63 MB/sec execute 314 sec latency 1908.080 ms 2024-04-27T02:18:07.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:07 smithi077 ceph-mon[50325]: pgmap v74: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.5 MiB/s wr, 261 op/s; 216/1620 objects degraded (13.333%) 2024-04-27T02:18:07.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:07 smithi077 ceph-mon[50325]: osdmap e78: 8 total, 8 up, 8 in 2024-04-27T02:18:07.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:07.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:07 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:08.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 193730 10.62 MB/sec execute 315 sec latency 1939.520 ms 2024-04-27T02:18:08.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:08 smithi008 ceph-mon[63900]: osdmap e79: 8 total, 8 up, 8 in 2024-04-27T02:18:08.979 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:08 smithi008 ceph-mon[63900]: pgmap v77: 97 pgs: 8 remapped+peering, 1 active+recovering+degraded, 5 active+undersized+degraded, 3 active+recovery_wait+degraded, 5 active+undersized, 75 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 516 op/s; 116/1629 objects degraded (7.121%); 22 B/s, 2 objects/s recovering 2024-04-27T02:18:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:08 smithi077 ceph-mon[50325]: osdmap e79: 8 total, 8 up, 8 in 2024-04-27T02:18:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:08 smithi077 ceph-mon[50325]: pgmap v77: 97 pgs: 8 remapped+peering, 1 active+recovering+degraded, 5 active+undersized+degraded, 3 active+recovery_wait+degraded, 5 active+undersized, 75 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 516 op/s; 116/1629 objects degraded (7.121%); 22 B/s, 2 objects/s recovering 2024-04-27T02:18:09.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194073 10.62 MB/sec execute 316 sec latency 696.201 ms 2024-04-27T02:18:09.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:09 smithi008 ceph-mon[63900]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T02:18:09.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:09 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 116/1629 objects degraded (7.121%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:09 smithi077 ceph-mon[50325]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T02:18:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:09 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 116/1629 objects degraded (7.121%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:10.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194468 10.62 MB/sec execute 317 sec latency 723.844 ms 2024-04-27T02:18:11.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:10 smithi008 ceph-mon[63900]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T02:18:11.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:10 smithi008 ceph-mon[63900]: pgmap v80: 97 pgs: 2 peering, 85 active+clean, 3 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 549 op/s; 64/1620 objects degraded (3.951%); 22 B/s, 25 objects/s recovering 2024-04-27T02:18:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:10 smithi077 ceph-mon[50325]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T02:18:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:10 smithi077 ceph-mon[50325]: pgmap v80: 97 pgs: 2 peering, 85 active+clean, 3 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 549 op/s; 64/1620 objects degraded (3.951%); 22 B/s, 25 objects/s recovering 2024-04-27T02:18:11.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 194868 10.60 MB/sec execute 318 sec latency 131.053 ms 2024-04-27T02:18:12.025 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:11 smithi008 ceph-mon[63900]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T02:18:12.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:11 smithi077 ceph-mon[50325]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T02:18:12.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195206 10.61 MB/sec execute 319 sec latency 946.530 ms 2024-04-27T02:18:13.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:12 smithi008 ceph-mon[63900]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T02:18:13.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:12 smithi008 ceph-mon[63900]: pgmap v83: 97 pgs: 2 peering, 85 active+clean, 3 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 625 KiB/s wr, 32 op/s; 64/1620 objects degraded (3.951%); 0 B/s, 23 objects/s recovering 2024-04-27T02:18:13.041 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:12 smithi008 ceph-mon[63900]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T02:18:13.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:12 smithi077 ceph-mon[50325]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T02:18:13.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:12 smithi077 ceph-mon[50325]: pgmap v83: 97 pgs: 2 peering, 85 active+clean, 3 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 625 KiB/s wr, 32 op/s; 64/1620 objects degraded (3.951%); 0 B/s, 23 objects/s recovering 2024-04-27T02:18:13.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:12 smithi077 ceph-mon[50325]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T02:18:13.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 195646 10.62 MB/sec execute 320 sec latency 967.728 ms 2024-04-27T02:18:14.323 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: pgmap v85: 97 pgs: 3 peering, 93 active+clean, 1 remapped+peering; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.4 MiB/s wr, 699 op/s; 3.6 MiB/s, 36 objects/s recovering 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: pgmap v86: 97 pgs: 3 peering, 93 active+clean, 1 remapped+peering; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 740 op/s; 3.8 MiB/s, 38 objects/s recovering 2024-04-27T02:18:14.324 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:14.325 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:14.326 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:14 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: pgmap v85: 97 pgs: 3 peering, 93 active+clean, 1 remapped+peering; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.4 MiB/s wr, 699 op/s; 3.6 MiB/s, 36 objects/s recovering 2024-04-27T02:18:14.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: pgmap v86: 97 pgs: 3 peering, 93 active+clean, 1 remapped+peering; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 740 op/s; 3.8 MiB/s, 38 objects/s recovering 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:14.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:14 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-27T02:18:14.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196029 10.63 MB/sec execute 321 sec latency 47.305 ms 2024-04-27T02:18:15.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:15 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/1620 objects degraded (3.951%), 4 pgs degraded) 2024-04-27T02:18:15.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:15 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/1620 objects degraded (3.951%), 4 pgs degraded) 2024-04-27T02:18:15.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196465 10.62 MB/sec execute 322 sec latency 61.423 ms 2024-04-27T02:18:16.350 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:16 smithi008 ceph-mon[63900]: pgmap v87: 97 pgs: 1 peering, 96 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 594 op/s; 2.8 MiB/s, 33 objects/s recovering 2024-04-27T02:18:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:16 smithi077 ceph-mon[50325]: pgmap v87: 97 pgs: 1 peering, 96 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 594 op/s; 2.8 MiB/s, 33 objects/s recovering 2024-04-27T02:18:16.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 196818 10.61 MB/sec execute 323 sec latency 65.649 ms 2024-04-27T02:18:17.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:17.357 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:17.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:17.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197224 10.63 MB/sec execute 324 sec latency 59.534 ms 2024-04-27T02:18:18.370 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:18 smithi008 ceph-mon[63900]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 823 op/s; 2.3 MiB/s, 26 objects/s recovering 2024-04-27T02:18:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:18 smithi077 ceph-mon[50325]: pgmap v88: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 8.9 MiB/s wr, 823 op/s; 2.3 MiB/s, 26 objects/s recovering 2024-04-27T02:18:18.875 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197613 10.61 MB/sec execute 325 sec latency 39.170 ms 2024-04-27T02:18:19.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 197941 10.60 MB/sec execute 326 sec latency 86.938 ms 2024-04-27T02:18:20.455 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:20 smithi008 ceph-mon[63900]: pgmap v89: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 618 op/s; 1.7 MiB/s, 19 objects/s recovering 2024-04-27T02:18:20.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:20 smithi077 ceph-mon[50325]: pgmap v89: 97 pgs: 97 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 618 op/s; 1.7 MiB/s, 19 objects/s recovering 2024-04-27T02:18:20.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198263 10.60 MB/sec execute 327 sec latency 45.757 ms 2024-04-27T02:18:21.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198585 10.59 MB/sec execute 328 sec latency 40.366 ms 2024-04-27T02:18:22.251 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:22 smithi008 ceph-mon[63900]: pgmap v90: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 405 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:18:22.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:22 smithi077 ceph-mon[50325]: pgmap v90: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 405 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:18:22.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 198926 10.59 MB/sec execute 329 sec latency 43.316 ms 2024-04-27T02:18:23.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199266 10.59 MB/sec execute 330 sec latency 41.493 ms 2024-04-27T02:18:24.299 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:24 smithi008 ceph-mon[63900]: pgmap v91: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 553 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:18:24.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:24 smithi077 ceph-mon[50325]: pgmap v91: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 553 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:18:24.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199536 10.59 MB/sec execute 331 sec latency 52.960 ms 2024-04-27T02:18:25.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 199760 10.57 MB/sec execute 332 sec latency 68.759 ms 2024-04-27T02:18:26.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:26 smithi008 ceph-mon[63900]: pgmap v92: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T02:18:26.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:26 smithi077 ceph-mon[50325]: pgmap v92: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T02:18:26.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200068 10.56 MB/sec execute 333 sec latency 48.499 ms 2024-04-27T02:18:27.347 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:27.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200308 10.54 MB/sec execute 334 sec latency 48.495 ms 2024-04-27T02:18:28.371 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:28 smithi008 ceph-mon[63900]: pgmap v93: 97 pgs: 97 active+clean; 352 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.9 MiB/s wr, 565 op/s 2024-04-27T02:18:28.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:28 smithi077 ceph-mon[50325]: pgmap v93: 97 pgs: 97 active+clean; 352 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.8 KiB/s rd, 5.9 MiB/s wr, 565 op/s 2024-04-27T02:18:28.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 200757 10.56 MB/sec execute 335 sec latency 51.838 ms 2024-04-27T02:18:29.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:29 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:29.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:29 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:29.384 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:29 smithi008 ceph-mon[63900]: Upgrade: osd.1 is safe to restart 2024-04-27T02:18:29.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:29 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:29.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:29 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T02:18:29.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:29 smithi077 ceph-mon[50325]: Upgrade: osd.1 is safe to restart 2024-04-27T02:18:29.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201258 10.56 MB/sec execute 336 sec latency 37.418 ms 2024-04-27T02:18:30.401 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:30 smithi008 ceph-mon[63900]: pgmap v94: 97 pgs: 97 active+clean; 354 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 476 op/s 2024-04-27T02:18:30.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:30 smithi077 ceph-mon[50325]: pgmap v94: 97 pgs: 97 active+clean; 354 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 476 op/s 2024-04-27T02:18:30.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 201693 10.55 MB/sec execute 337 sec latency 37.591 ms 2024-04-27T02:18:31.430 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:31.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:31.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202296 10.58 MB/sec execute 338 sec latency 41.876 ms 2024-04-27T02:18:32.462 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: pgmap v95: 97 pgs: 97 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 484 op/s 2024-04-27T02:18:32.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: Upgrade: Updating osd.1 2024-04-27T02:18:32.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:32.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:18:32.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:18:32.463 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:32 smithi008 ceph-mon[63900]: Deploying daemon osd.1 on smithi008 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: pgmap v95: 97 pgs: 97 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 484 op/s 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: Upgrade: Updating osd.1 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:18:32.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:32 smithi077 ceph-mon[50325]: Deploying daemon osd.1 on smithi008 2024-04-27T02:18:32.883 INFO:teuthology.orchestra.run.smithi008.stdout: 5 202839 10.60 MB/sec execute 339 sec latency 30.054 ms 2024-04-27T02:18:33.454 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:18:33.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203190 10.60 MB/sec execute 340 sec latency 32.245 ms 2024-04-27T02:18:34.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:34 smithi008 ceph-mon[63900]: pgmap v96: 97 pgs: 97 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 638 op/s 2024-04-27T02:18:34.485 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:34 smithi008 ceph-mon[63900]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 28s ago 14m 16.4M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (79s) 28s ago 14m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (73s) 68s ago 13m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 28s ago 13m - - 2024-04-27T02:18:34.562 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (8m) 28s ago 8m 40.6M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (8m) 68s ago 8m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (4m) 28s ago 15m 502M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (3m) 68s ago 13m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (2m) 28s ago 15m 48.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (2m) 68s ago 13m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (7m) 28s ago 7m 272M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (7m) 68s ago 7m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (4m) 28s ago 13m 10.8M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (4m) 68s ago 13m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (38s) 28s ago 12m 12.7M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (12m) 28s ago 12m 583M 1937M 17.2.0 e1d6a67b021e f43d363f6000 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (11m) 28s ago 11m 485M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 28s ago 11m 595M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:18:34.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (10m) 68s ago 10m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:18:34.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (10m) 68s ago 10m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:18:34.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (10m) 68s ago 10m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:18:34.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (9m) 68s ago 9m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:18:34.564 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 28s ago 12m 60.7M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:18:34.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:34 smithi077 ceph-mon[50325]: pgmap v96: 97 pgs: 97 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 638 op/s 2024-04-27T02:18:34.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:34 smithi077 ceph-mon[50325]: from='client.44159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:34.886 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203483 10.59 MB/sec execute 341 sec latency 63.552 ms 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:18:35.255 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:18:35.256 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:18:35.499 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:35 smithi008 ceph-mon[63900]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:35.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:35 smithi008 ceph-mon[63900]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:35.500 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:35 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/683759683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:35.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:35 smithi077 ceph-mon[50325]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:35.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:35 smithi077 ceph-mon[50325]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:35.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:35 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/683759683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "7/23 daemons upgraded", 2024-04-27T02:18:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:18:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:18:35.877 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:18:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 203752 10.57 MB/sec execute 342 sec latency 52.135 ms 2024-04-27T02:18:36.454 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:18:36.454 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:18:36.454 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:18:36.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:36 smithi008 ceph-mon[63900]: pgmap v97: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 476 op/s 2024-04-27T02:18:36.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:36 smithi077 ceph-mon[50325]: pgmap v97: 97 pgs: 97 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 476 op/s 2024-04-27T02:18:36.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204117 10.58 MB/sec execute 343 sec latency 41.956 ms 2024-04-27T02:18:37.516 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:37 smithi008 ceph-mon[63900]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:37.517 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:37 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3807061444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:18:37.517 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:37 smithi008 systemd[1]: Stopping Ceph osd.1 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:18:37.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:37 smithi077 ceph-mon[50325]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:18:37.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:37 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3807061444' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:18:37.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204611 10.58 MB/sec execute 344 sec latency 35.088 ms 2024-04-27T02:18:38.018 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:37 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[36412]: 2024-04-27T02:18:37.857+0000 7fdfd3ffe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:18:38.018 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:37 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[36412]: 2024-04-27T02:18:37.857+0000 7fdfd3ffe700 -1 osd.1 84 *** Got signal Terminated *** 2024-04-27T02:18:38.018 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:37 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[36412]: 2024-04-27T02:18:37.857+0000 7fdfd3ffe700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:18:38.636 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:38 smithi008 ceph-mon[63900]: pgmap v98: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 663 op/s 2024-04-27T02:18:38.636 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:38 smithi008 ceph-mon[63900]: osd.1 marked itself down and dead 2024-04-27T02:18:38.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:38 smithi077 ceph-mon[50325]: pgmap v98: 97 pgs: 97 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 663 op/s 2024-04-27T02:18:38.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:38 smithi077 ceph-mon[50325]: osd.1 marked itself down and dead 2024-04-27T02:18:38.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 204956 10.57 MB/sec execute 345 sec latency 540.429 ms 2024-04-27T02:18:39.343 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:39 smithi008 podman[72229]: 2024-04-27 02:18:39.008334546 +0000 UTC m=+1.298651708 container died f43d363f6000553bf031e448f4912e405a3402fbf17d6e485e7e8b8d405bf3b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, vcs-type=git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8, CEPH_POINT_RELEASE=-17.2.0, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD) 2024-04-27T02:18:39.597 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:39 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:18:39.597 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:39 smithi008 ceph-mon[63900]: osdmap e85: 8 total, 7 up, 8 in 2024-04-27T02:18:39.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:39 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:18:39.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:39 smithi077 ceph-mon[50325]: osdmap e85: 8 total, 7 up, 8 in 2024-04-27T02:18:39.851 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:39 smithi008 podman[72229]: 2024-04-27 02:18:39.636513103 +0000 UTC m=+1.926830266 container cleanup f43d363f6000553bf031e448f4912e405a3402fbf17d6e485e7e8b8d405bf3b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, version=8, vcs-type=git, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, release=754) 2024-04-27T02:18:39.851 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:39 smithi008 bash[72229]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1 2024-04-27T02:18:39.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205158 10.55 MB/sec execute 346 sec latency 1020.684 ms 2024-04-27T02:18:40.102 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:40 smithi008 podman[72247]: 2024-04-27 02:18:40.020794543 +0000 UTC m=+1.000920454 container remove f43d363f6000553bf031e448f4912e405a3402fbf17d6e485e7e8b8d405bf3b9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, version=8) 2024-04-27T02:18:40.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:40 smithi008 ceph-mon[63900]: pgmap v100: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 598 op/s 2024-04-27T02:18:40.353 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:40 smithi008 ceph-mon[63900]: osdmap e86: 8 total, 7 up, 8 in 2024-04-27T02:18:40.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:40 smithi077 ceph-mon[50325]: pgmap v100: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.9 MiB/s wr, 598 op/s 2024-04-27T02:18:40.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:40 smithi077 ceph-mon[50325]: osdmap e86: 8 total, 7 up, 8 in 2024-04-27T02:18:40.859 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:40 smithi008 podman[72333]: 2024-04-27 02:18:40.533832765 +0000 UTC m=+0.030013008 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:18:40.859 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:40 smithi008 podman[72333]: 2024-04-27 02:18:40.678403563 +0000 UTC m=+0.174583847 container create 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T02:18:40.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 205641 10.56 MB/sec execute 347 sec latency 27.557 ms 2024-04-27T02:18:41.113 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:41 smithi008 podman[72333]: 2024-04-27 02:18:41.029625935 +0000 UTC m=+0.525806174 container init 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T02:18:41.113 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:41 smithi008 podman[72333]: 2024-04-27 02:18:41.039592217 +0000 UTC m=+0.535772449 container start 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:41 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 18/1587 objects degraded (1.134%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:41.615 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:41 smithi008 podman[72333]: 2024-04-27 02:18:41.388318687 +0000 UTC m=+0.884498922 container attach 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T02:18:41.616 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:41 smithi008 podman[72333]: 2024-04-27 02:18:41.506791071 +0000 UTC m=+1.002971301 container died 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T02:18:41.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:41 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 18/1587 objects degraded (1.134%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:41.884 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206129 10.57 MB/sec execute 348 sec latency 30.163 ms 2024-04-27T02:18:42.633 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:42 smithi008 podman[72357]: 2024-04-27 02:18:42.50628326 +0000 UTC m=+0.992062107 container remove 35c71b4baf2112ae88fc549bc68cd97278b91d295187eec23f6a9f6fa5894c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-27T02:18:42.633 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:42 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-27T02:18:42.633 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:42 smithi008 systemd[1]: Stopped Ceph osd.1 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:18:42.633 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:42 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.1.service: Consumed 3min 50.272s CPU time. 2024-04-27T02:18:42.633 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:42 smithi008 ceph-mon[63900]: pgmap v102: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 12 stale+active+clean, 81 active+clean; 350 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 364 op/s; 18/1587 objects degraded (1.134%) 2024-04-27T02:18:42.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:42 smithi077 ceph-mon[50325]: pgmap v102: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 12 stale+active+clean, 81 active+clean; 350 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 364 op/s; 18/1587 objects degraded (1.134%) 2024-04-27T02:18:42.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206535 10.58 MB/sec execute 349 sec latency 33.031 ms 2024-04-27T02:18:43.154 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:42 smithi008 systemd[1]: Starting Ceph osd.1 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:18:43.408 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:43 smithi008 podman[72454]: 2024-04-27 02:18:43.082312567 +0000 UTC m=+0.023570391 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:18:43.658 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:43 smithi008 podman[72454]: 2024-04-27 02:18:43.664934065 +0000 UTC m=+0.606191894 container create 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-27T02:18:43.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 206844 10.57 MB/sec execute 350 sec latency 42.589 ms 2024-04-27T02:18:44.173 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 podman[72454]: 2024-04-27 02:18:44.155728712 +0000 UTC m=+1.096986537 container init 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T02:18:44.174 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 podman[72454]: 2024-04-27 02:18:44.164104245 +0000 UTC m=+1.105362072 container start 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-27T02:18:44.428 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-mon[63900]: pgmap v103: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 665 op/s; 188/1659 objects degraded (11.332%) 2024-04-27T02:18:44.428 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 podman[72454]: 2024-04-27 02:18:44.327684435 +0000 UTC m=+1.268942266 container attach 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T02:18:44.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:44 smithi077 ceph-mon[50325]: pgmap v103: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 665 op/s; 188/1659 objects degraded (11.332%) 2024-04-27T02:18:44.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207174 10.56 MB/sec execute 351 sec latency 56.580 ms 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 bash[72454]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-27T02:18:44.935 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T02:18:44.936 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate[72468]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-27T02:18:44.936 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:44 smithi008 podman[72454]: 2024-04-27 02:18:44.824525032 +0000 UTC m=+1.765782859 container died 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T02:18:45.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207404 10.55 MB/sec execute 352 sec latency 63.057 ms 2024-04-27T02:18:46.222 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:45 smithi008 podman[72599]: 2024-04-27 02:18:45.995262539 +0000 UTC m=+1.165075869 container remove 264268e3f0e629e1dec6fc584381f2373fb75c9c25c22be6af4b6f98339d2a27 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-27T02:18:46.222 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 podman[72664]: 2024-04-27 02:18:46.138167345 +0000 UTC m=+0.020763506 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:18:46.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:46 smithi008 ceph-mon[63900]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.3 MiB/s wr, 385 op/s; 187/1689 objects degraded (11.072%) 2024-04-27T02:18:46.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:46.473 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:46.473 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 podman[72664]: 2024-04-27 02:18:46.299115034 +0000 UTC m=+0.181711164 container create 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-27T02:18:46.724 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 podman[72664]: 2024-04-27 02:18:46.59732024 +0000 UTC m=+0.479916371 container init 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-27T02:18:46.724 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 podman[72664]: 2024-04-27 02:18:46.608538244 +0000 UTC m=+0.491134384 container start 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb) 2024-04-27T02:18:46.724 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 ceph-osd[72682]: -- 172.21.15.8:0/1030570234 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560790287680 con 0x560790250c00 2024-04-27T02:18:46.724 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 bash[72664]: 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 2024-04-27T02:18:46.724 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:46 smithi008 systemd[1]: Started Ceph osd.1 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:18:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:46 smithi077 ceph-mon[50325]: pgmap v104: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 347 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.3 MiB/s wr, 385 op/s; 187/1689 objects degraded (11.072%) 2024-04-27T02:18:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:18:46.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207647 10.54 MB/sec execute 353 sec latency 60.899 ms 2024-04-27T02:18:47.232 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:47 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:18:47.188+0000 7f37bdbbd7c0 -1 Falling back to public interface 2024-04-27T02:18:47.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 207873 10.53 MB/sec execute 354 sec latency 93.369 ms 2024-04-27T02:18:47.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:47.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:47.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:18:48.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:48.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:48.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:18:48.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:48 smithi008 ceph-mon[63900]: pgmap v105: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s; 180/1599 objects degraded (11.257%) 2024-04-27T02:18:48.788 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:48 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 180/1599 objects degraded (11.257%), 24 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:48.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208153 10.52 MB/sec execute 355 sec latency 90.576 ms 2024-04-27T02:18:49.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:48 smithi077 ceph-mon[50325]: pgmap v105: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s; 180/1599 objects degraded (11.257%) 2024-04-27T02:18:49.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:48 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 180/1599 objects degraded (11.257%), 24 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:49.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208479 10.51 MB/sec execute 356 sec latency 48.308 ms 2024-04-27T02:18:50.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:50 smithi008 ceph-mon[63900]: pgmap v106: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 515 op/s; 180/1599 objects degraded (11.257%) 2024-04-27T02:18:50.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 208819 10.50 MB/sec execute 357 sec latency 37.270 ms 2024-04-27T02:18:51.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:50 smithi077 ceph-mon[50325]: pgmap v106: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 350 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 515 op/s; 180/1599 objects degraded (11.257%) 2024-04-27T02:18:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209169 10.50 MB/sec execute 358 sec latency 39.958 ms 2024-04-27T02:18:52.642 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:52 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:18:52.624+0000 7f37bdbbd7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-27T02:18:52.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209491 10.50 MB/sec execute 359 sec latency 43.636 ms 2024-04-27T02:18:52.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:52 smithi008 ceph-mon[63900]: pgmap v107: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 450 op/s; 179/1584 objects degraded (11.301%) 2024-04-27T02:18:53.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:52 smithi077 ceph-mon[50325]: pgmap v107: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 349 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 450 op/s; 179/1584 objects degraded (11.301%) 2024-04-27T02:18:53.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 209833 10.50 MB/sec execute 360 sec latency 51.059 ms 2024-04-27T02:18:53.952 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:53 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 185/1608 objects degraded (11.505%), 24 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:54.203 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:18:54.087+0000 7f37bdbbd7c0 -1 osd.1 84 log_to_monitors true 2024-04-27T02:18:54.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:53 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 185/1608 objects degraded (11.505%), 24 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:54.893 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210194 10.51 MB/sec execute 361 sec latency 41.410 ms 2024-04-27T02:18:54.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:54 smithi008 ceph-mon[63900]: pgmap v108: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 351 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 599 op/s; 185/1608 objects degraded (11.505%) 2024-04-27T02:18:54.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:54 smithi008 ceph-mon[63900]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:18:54.956 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:18:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:18:54.871+0000 7f37af015700 -1 osd.1 84 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:18:55.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:54 smithi077 ceph-mon[50325]: pgmap v108: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 351 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.8 MiB/s wr, 599 op/s; 185/1608 objects degraded (11.505%) 2024-04-27T02:18:55.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:54 smithi077 ceph-mon[50325]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T02:18:55.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210487 10.50 MB/sec execute 362 sec latency 66.788 ms 2024-04-27T02:18:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:55 smithi008 ceph-mon[63900]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:18:55.984 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:55 smithi008 ceph-mon[63900]: osdmap e87: 8 total, 7 up, 8 in 2024-04-27T02:18:55.985 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:55 smithi008 ceph-mon[63900]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:18:55.985 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:55.985 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:55 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:55 smithi077 ceph-mon[50325]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T02:18:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:55 smithi077 ceph-mon[50325]: osdmap e87: 8 total, 7 up, 8 in 2024-04-27T02:18:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:55 smithi077 ceph-mon[50325]: from='osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:18:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:56.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:55 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:56.892 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210570 10.47 MB/sec execute 363 sec latency 1037.610 ms 2024-04-27T02:18:57.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: pgmap v110: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.6 MiB/s wr, 454 op/s; 183/1608 objects degraded (11.381%) 2024-04-27T02:18:57.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:18:57.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903] boot 2024-04-27T02:18:57.065 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T02:18:57.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:18:57.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:57.066 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: pgmap v110: 97 pgs: 15 active+undersized, 24 active+undersized+degraded, 58 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.6 MiB/s wr, 454 op/s; 183/1608 objects degraded (11.381%) 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: osd.1 [v2:172.21.15.8:6810/1995207903,v1:172.21.15.8:6811/1995207903] boot 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:57.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:18:57.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:57 smithi008 ceph-mon[63900]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T02:18:57.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:57 smithi008 ceph-mon[63900]: pgmap v113: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering, 6 active+undersized, 7 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 682 op/s; 60/1587 objects degraded (3.781%); 0 B/s, 4 objects/s recovering 2024-04-27T02:18:57.893 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210652 10.44 MB/sec execute 364 sec latency 1980.109 ms 2024-04-27T02:18:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:57 smithi077 ceph-mon[50325]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T02:18:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:57 smithi077 ceph-mon[50325]: pgmap v113: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering, 6 active+undersized, 7 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 682 op/s; 60/1587 objects degraded (3.781%); 0 B/s, 4 objects/s recovering 2024-04-27T02:18:58.861 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:58 smithi008 ceph-mon[63900]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T02:18:58.893 INFO:teuthology.orchestra.run.smithi008.stdout: 5 210969 10.43 MB/sec execute 365 sec latency 913.015 ms 2024-04-27T02:18:59.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:58 smithi077 ceph-mon[50325]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T02:18:59.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:59 smithi008 ceph-mon[63900]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T02:18:59.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:59 smithi008 ceph-mon[63900]: pgmap v116: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering, 6 active+undersized, 7 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 462 op/s; 60/1587 objects degraded (3.781%); 0 B/s, 6 objects/s recovering 2024-04-27T02:18:59.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:59 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 60/1587 objects degraded (3.781%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T02:18:59.880 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:18:59 smithi008 ceph-mon[63900]: osdmap e92: 8 total, 8 up, 8 in 2024-04-27T02:18:59.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211182 10.42 MB/sec execute 366 sec latency 525.765 ms 2024-04-27T02:19:00.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:59 smithi077 ceph-mon[50325]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T02:19:00.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:59 smithi077 ceph-mon[50325]: pgmap v116: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 7 remapped+peering, 6 active+undersized, 7 active+undersized+degraded, 75 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 462 op/s; 60/1587 objects degraded (3.781%); 0 B/s, 6 objects/s recovering 2024-04-27T02:19:00.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:59 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 60/1587 objects degraded (3.781%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:00.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:18:59 smithi077 ceph-mon[50325]: osdmap e92: 8 total, 8 up, 8 in 2024-04-27T02:19:00.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211465 10.42 MB/sec execute 367 sec latency 675.050 ms 2024-04-27T02:19:01.495 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:01 smithi008 ceph-mon[63900]: osdmap e93: 8 total, 8 up, 8 in 2024-04-27T02:19:01.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:01.496 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:01 smithi077 ceph-mon[50325]: osdmap e93: 8 total, 8 up, 8 in 2024-04-27T02:19:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:01.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 211814 10.41 MB/sec execute 368 sec latency 409.908 ms 2024-04-27T02:19:02.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: pgmap v119: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 694 KiB/s wr, 38 op/s; 19/1536 objects degraded (1.237%); 1.1 MiB/s, 69 keys/s, 11 objects/s recovering 2024-04-27T02:19:02.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: osdmap e94: 8 total, 8 up, 8 in 2024-04-27T02:19:02.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.691 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: pgmap v121: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 960 KiB/s wr, 53 op/s; 19/1536 objects degraded (1.237%); 1.5 MiB/s, 96 keys/s, 16 objects/s recovering 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: pgmap v122: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s wr, 62 op/s; 19/1536 objects degraded (1.237%); 1.7 MiB/s, 110 keys/s, 19 objects/s recovering 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.692 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: pgmap v119: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 694 KiB/s wr, 38 op/s; 19/1536 objects degraded (1.237%); 1.1 MiB/s, 69 keys/s, 11 objects/s recovering 2024-04-27T02:19:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: osdmap e94: 8 total, 8 up, 8 in 2024-04-27T02:19:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: pgmap v121: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 960 KiB/s wr, 53 op/s; 19/1536 objects degraded (1.237%); 1.5 MiB/s, 96 keys/s, 16 objects/s recovering 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: pgmap v122: 97 pgs: 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 5 remapped+peering, 88 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 MiB/s wr, 62 op/s; 19/1536 objects degraded (1.237%); 1.7 MiB/s, 110 keys/s, 19 objects/s recovering 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:02.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:02.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212264 10.41 MB/sec execute 369 sec latency 292.084 ms 2024-04-27T02:19:03.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:03 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:03.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:03 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-27T02:19:03.725 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:03 smithi008 ceph-mon[63900]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T02:19:03.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:03 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:03.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:03 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-27T02:19:03.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:03 smithi077 ceph-mon[50325]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T02:19:03.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212501 10.40 MB/sec execute 370 sec latency 906.772 ms 2024-04-27T02:19:04.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:04 smithi008 ceph-mon[63900]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T02:19:04.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:04 smithi008 ceph-mon[63900]: pgmap v125: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 93 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 715 op/s; 1/1581 objects degraded (0.063%); 0 B/s, 53 objects/s recovering 2024-04-27T02:19:04.488 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:04 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 19/1536 objects degraded (1.237%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:04.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 212833 10.41 MB/sec execute 371 sec latency 60.351 ms 2024-04-27T02:19:04.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:04 smithi077 ceph-mon[50325]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T02:19:04.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:04 smithi077 ceph-mon[50325]: pgmap v125: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 93 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 715 op/s; 1/1581 objects degraded (0.063%); 0 B/s, 53 objects/s recovering 2024-04-27T02:19:04.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:04 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 19/1536 objects degraded (1.237%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:05.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213339 10.41 MB/sec execute 372 sec latency 35.867 ms 2024-04-27T02:19:06.053 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:05 smithi008 ceph-mon[63900]: pgmap v126: 97 pgs: 1 peering, 96 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 488 op/s; 0 B/s, 36 objects/s recovering 2024-04-27T02:19:06.053 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:05 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1581 objects degraded (0.063%), 1 pg degraded) 2024-04-27T02:19:06.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:05 smithi077 ceph-mon[50325]: pgmap v126: 97 pgs: 1 peering, 96 active+clean; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 488 op/s; 0 B/s, 36 objects/s recovering 2024-04-27T02:19:06.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:05 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1581 objects degraded (0.063%), 1 pg degraded) 2024-04-27T02:19:06.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213652 10.42 MB/sec execute 373 sec latency 57.193 ms 2024-04-27T02:19:07.067 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:19:07.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 213962 10.41 MB/sec execute 374 sec latency 53.043 ms 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (5m) 12s ago 14m 16.4M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (113s) 12s ago 14m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (107s) 102s ago 13m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 12s ago 14m - - 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (8m) 12s ago 8m 40.9M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (8m) 102s ago 8m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (4m) 12s ago 15m 505M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (3m) 102s ago 13m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:19:08.214 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (3m) 12s ago 16m 51.1M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (2m) 102s ago 13m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (8m) 12s ago 8m 279M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (8m) 102s ago 8m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 12s ago 14m 10.9M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (4m) 102s ago 13m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (72s) 12s ago 13m 256M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (22s) 12s ago 12m 12.3M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (12m) 12s ago 12m 533M 1937M 17.2.0 e1d6a67b021e 7852c8ab547b 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (11m) 12s ago 11m 376M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (11m) 102s ago 11m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (10m) 102s ago 10m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (10m) 102s ago 10m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (10m) 102s ago 10m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:19:08.215 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (3m) 12s ago 13m 63.0M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:19:08.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:08 smithi008 ceph-mon[63900]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:08.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:08 smithi008 ceph-mon[63900]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:08.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:08 smithi008 ceph-mon[63900]: pgmap v127: 97 pgs: 97 active+clean; 357 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 727 op/s; 0 B/s, 26 objects/s recovering 2024-04-27T02:19:08.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214335 10.40 MB/sec execute 375 sec latency 43.635 ms 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:19:08.912 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 6 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:19:08.913 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:19:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:08 smithi077 ceph-mon[50325]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:08 smithi077 ceph-mon[50325]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:09.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:08 smithi077 ceph-mon[50325]: pgmap v127: 97 pgs: 97 active+clean; 357 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 727 op/s; 0 B/s, 26 objects/s recovering 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:19:09.482 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "8/23 daemons upgraded", 2024-04-27T02:19:09.483 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:19:09.483 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:19:09.483 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:19:09.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:09 smithi008 ceph-mon[63900]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:09.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:09 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2271086523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:09.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214675 10.39 MB/sec execute 376 sec latency 44.904 ms 2024-04-27T02:19:10.171 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:19:10.171 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:19:10.171 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:19:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:09 smithi077 ceph-mon[50325]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:09 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2271086523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:10.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:10 smithi008 ceph-mon[63900]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:10.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:10 smithi008 ceph-mon[63900]: pgmap v128: 97 pgs: 97 active+clean; 358 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 568 op/s; 0 B/s, 19 objects/s recovering 2024-04-27T02:19:10.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:10 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1778723587' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:19:10.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 214982 10.39 MB/sec execute 377 sec latency 65.680 ms 2024-04-27T02:19:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:10 smithi077 ceph-mon[50325]: from='client.44199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:10 smithi077 ceph-mon[50325]: pgmap v128: 97 pgs: 97 active+clean; 358 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 568 op/s; 0 B/s, 19 objects/s recovering 2024-04-27T02:19:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:10 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1778723587' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:19:11.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215475 10.39 MB/sec execute 378 sec latency 41.187 ms 2024-04-27T02:19:12.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 215889 10.39 MB/sec execute 379 sec latency 49.778 ms 2024-04-27T02:19:12.905 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:12 smithi008 ceph-mon[63900]: pgmap v129: 97 pgs: 97 active+clean; 359 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.0 MiB/s wr, 385 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:19:13.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:12 smithi077 ceph-mon[50325]: pgmap v129: 97 pgs: 97 active+clean; 359 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.0 MiB/s wr, 385 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:19:13.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216206 10.39 MB/sec execute 380 sec latency 46.968 ms 2024-04-27T02:19:14.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 216684 10.41 MB/sec execute 381 sec latency 44.876 ms 2024-04-27T02:19:14.937 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:14 smithi008 ceph-mon[63900]: pgmap v130: 97 pgs: 97 active+clean; 353 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 481 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:19:15.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:14 smithi077 ceph-mon[50325]: pgmap v130: 97 pgs: 97 active+clean; 353 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 481 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T02:19:15.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217073 10.41 MB/sec execute 382 sec latency 38.166 ms 2024-04-27T02:19:16.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217319 10.40 MB/sec execute 383 sec latency 40.584 ms 2024-04-27T02:19:16.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:16 smithi008 ceph-mon[63900]: pgmap v131: 97 pgs: 97 active+clean; 352 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 464 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T02:19:16.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:16.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:16.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:17.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:16 smithi077 ceph-mon[50325]: pgmap v131: 97 pgs: 97 active+clean; 352 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 464 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T02:19:17.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:17.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:17.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:17.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 217655 10.40 MB/sec execute 384 sec latency 46.471 ms 2024-04-27T02:19:17.986 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:18.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:18.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218052 10.38 MB/sec execute 385 sec latency 61.793 ms 2024-04-27T02:19:18.989 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:18 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:18.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:18 smithi008 ceph-mon[63900]: Upgrade: osd.2 is safe to restart 2024-04-27T02:19:18.990 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:18 smithi008 ceph-mon[63900]: pgmap v132: 97 pgs: 97 active+clean; 353 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 641 op/s 2024-04-27T02:19:19.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:18 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T02:19:19.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:18 smithi077 ceph-mon[50325]: Upgrade: osd.2 is safe to restart 2024-04-27T02:19:19.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:18 smithi077 ceph-mon[50325]: pgmap v132: 97 pgs: 97 active+clean; 353 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.3 MiB/s wr, 641 op/s 2024-04-27T02:19:19.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218387 10.39 MB/sec execute 386 sec latency 42.991 ms 2024-04-27T02:19:20.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 218708 10.38 MB/sec execute 387 sec latency 76.245 ms 2024-04-27T02:19:21.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:20 smithi008 ceph-mon[63900]: pgmap v133: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 464 op/s 2024-04-27T02:19:21.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:21.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:19:21.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:21.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:20 smithi077 ceph-mon[50325]: pgmap v133: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 464 op/s 2024-04-27T02:19:21.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:21.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T02:19:21.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:21.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219094 10.38 MB/sec execute 388 sec latency 67.943 ms 2024-04-27T02:19:22.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:21 smithi008 ceph-mon[63900]: Upgrade: Updating osd.2 2024-04-27T02:19:22.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:21 smithi008 ceph-mon[63900]: Deploying daemon osd.2 on smithi008 2024-04-27T02:19:22.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:21 smithi077 ceph-mon[50325]: Upgrade: Updating osd.2 2024-04-27T02:19:22.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:21 smithi077 ceph-mon[50325]: Deploying daemon osd.2 on smithi008 2024-04-27T02:19:22.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219459 10.37 MB/sec execute 389 sec latency 38.658 ms 2024-04-27T02:19:23.054 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:22 smithi008 ceph-mon[63900]: pgmap v134: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 456 op/s 2024-04-27T02:19:23.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:22 smithi077 ceph-mon[50325]: pgmap v134: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 456 op/s 2024-04-27T02:19:23.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 219730 10.36 MB/sec execute 390 sec latency 54.788 ms 2024-04-27T02:19:24.829 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:24 smithi008 ceph-mon[63900]: pgmap v135: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-04-27T02:19:24.899 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220029 10.37 MB/sec execute 391 sec latency 61.479 ms 2024-04-27T02:19:25.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:24 smithi077 ceph-mon[50325]: pgmap v135: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-04-27T02:19:25.853 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:25 smithi008 ceph-mon[63900]: pgmap v136: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-04-27T02:19:25.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220401 10.37 MB/sec execute 392 sec latency 47.838 ms 2024-04-27T02:19:26.111 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:25 smithi008 systemd[1]: Stopping Ceph osd.2 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:19:26.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:25 smithi077 ceph-mon[50325]: pgmap v136: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-04-27T02:19:26.363 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:26 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[40123]: 2024-04-27T02:19:26.269+0000 7f2bc859d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:19:26.363 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:26 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[40123]: 2024-04-27T02:19:26.269+0000 7f2bc859d700 -1 osd.2 96 *** Got signal Terminated *** 2024-04-27T02:19:26.363 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:26 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[40123]: 2024-04-27T02:19:26.269+0000 7f2bc859d700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:19:26.865 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:26 smithi008 ceph-mon[63900]: osd.2 marked itself down and dead 2024-04-27T02:19:26.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220687 10.36 MB/sec execute 393 sec latency 72.290 ms 2024-04-27T02:19:27.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:26 smithi077 ceph-mon[50325]: osd.2 marked itself down and dead 2024-04-27T02:19:27.638 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:27 smithi008 podman[74775]: 2024-04-27 02:19:27.532047183 +0000 UTC m=+1.432364106 container died 7852c8ab547b2a9d48f783ad310feb8d61ad1ba6624c5933f95409395a753b05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, vcs-type=git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, version=8, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream) 2024-04-27T02:19:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 220811 10.34 MB/sec execute 394 sec latency 1018.803 ms 2024-04-27T02:19:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:27 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:19:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:27 smithi008 ceph-mon[63900]: osdmap e97: 8 total, 7 up, 8 in 2024-04-27T02:19:27.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:27 smithi008 ceph-mon[63900]: pgmap v138: 97 pgs: 17 peering, 80 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 493 op/s 2024-04-27T02:19:28.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:27 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:19:28.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:27 smithi077 ceph-mon[50325]: osdmap e97: 8 total, 7 up, 8 in 2024-04-27T02:19:28.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:27 smithi077 ceph-mon[50325]: pgmap v138: 97 pgs: 17 peering, 80 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 493 op/s 2024-04-27T02:19:28.426 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:28 smithi008 podman[74775]: 2024-04-27 02:19:28.22402568 +0000 UTC m=+2.124342607 container cleanup 7852c8ab547b2a9d48f783ad310feb8d61ad1ba6624c5933f95409395a753b05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, name=centos-stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-27T02:19:28.426 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:28 smithi008 bash[74775]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2 2024-04-27T02:19:28.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221033 10.33 MB/sec execute 395 sec latency 104.415 ms 2024-04-27T02:19:28.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:28 smithi008 ceph-mon[63900]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:19:28.948 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:28 smithi008 ceph-mon[63900]: osdmap e98: 8 total, 7 up, 8 in 2024-04-27T02:19:28.948 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:28 smithi008 podman[74791]: 2024-04-27 02:19:28.952821169 +0000 UTC m=+1.415441897 container remove 7852c8ab547b2a9d48f783ad310feb8d61ad1ba6624c5933f95409395a753b05 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., release=754, io.buildah.version=1.19.8, vcs-type=git, GIT_CLEAN=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8) 2024-04-27T02:19:29.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:28 smithi077 ceph-mon[50325]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:19:29.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:28 smithi077 ceph-mon[50325]: osdmap e98: 8 total, 7 up, 8 in 2024-04-27T02:19:29.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:29 smithi008 podman[74878]: 2024-04-27 02:19:29.310855305 +0000 UTC m=+0.025164132 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:19:29.457 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:29 smithi008 podman[74878]: 2024-04-27 02:19:29.437674207 +0000 UTC m=+0.151983007 container create 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:19:29.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221309 10.33 MB/sec execute 396 sec latency 56.217 ms 2024-04-27T02:19:29.997 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:29 smithi008 ceph-mon[63900]: pgmap v140: 97 pgs: 1 active+undersized, 1 active+undersized+degraded, 3 stale+active+clean, 17 peering, 75 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 601 op/s; 14/1608 objects degraded (0.871%) 2024-04-27T02:19:29.997 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:29 smithi008 podman[74878]: 2024-04-27 02:19:29.793658455 +0000 UTC m=+0.507967264 container init 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T02:19:29.997 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:29 smithi008 podman[74878]: 2024-04-27 02:19:29.805395212 +0000 UTC m=+0.519704020 container start 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T02:19:29.997 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:29 smithi008 podman[74878]: 2024-04-27 02:19:29.909374997 +0000 UTC m=+0.623683813 container attach 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:19:30.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:29 smithi077 ceph-mon[50325]: pgmap v140: 97 pgs: 1 active+undersized, 1 active+undersized+degraded, 3 stale+active+clean, 17 peering, 75 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 601 op/s; 14/1608 objects degraded (0.871%) 2024-04-27T02:19:30.247 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:30 smithi008 podman[74878]: 2024-04-27 02:19:30.245155312 +0000 UTC m=+0.959464120 container died 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T02:19:30.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221671 10.32 MB/sec execute 397 sec latency 38.009 ms 2024-04-27T02:19:31.006 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:30 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 14/1608 objects degraded (0.871%), 1 pg degraded (PG_DEGRADED) 2024-04-27T02:19:31.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:30 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 14/1608 objects degraded (0.871%), 1 pg degraded (PG_DEGRADED) 2024-04-27T02:19:31.258 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 podman[74904]: 2024-04-27 02:19:31.187481556 +0000 UTC m=+0.936309065 container remove 0f3a9027f901791326d5a193a508e5d2432d8a106bd21a302880e881833891d2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:19:31.258 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-27T02:19:31.258 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 systemd[1]: Stopped Ceph osd.2 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:19:31.258 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.2.service: Consumed 3min 3.734s CPU time. 2024-04-27T02:19:31.517 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 systemd[1]: Starting Ceph osd.2 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:19:31.768 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 podman[75005]: 2024-04-27 02:19:31.662578913 +0000 UTC m=+0.023662353 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:19:31.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 221916 10.31 MB/sec execute 398 sec latency 68.206 ms 2024-04-27T02:19:32.033 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:31 smithi008 podman[75005]: 2024-04-27 02:19:31.794577237 +0000 UTC m=+0.155660676 container create f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:19:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:32.284 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-mon[63900]: pgmap v141: 97 pgs: 3 active+undersized, 5 active+undersized+degraded, 1 stale+active+clean, 17 peering, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 309 op/s; 50/1602 objects degraded (3.121%) 2024-04-27T02:19:32.284 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 podman[75005]: 2024-04-27 02:19:32.251397744 +0000 UTC m=+0.612481184 container init f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-27T02:19:32.284 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 podman[75005]: 2024-04-27 02:19:32.257248798 +0000 UTC m=+0.618332247 container start f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T02:19:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:32 smithi077 ceph-mon[50325]: pgmap v141: 97 pgs: 3 active+undersized, 5 active+undersized+degraded, 1 stale+active+clean, 17 peering, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 309 op/s; 50/1602 objects degraded (3.121%) 2024-04-27T02:19:32.553 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 podman[75005]: 2024-04-27 02:19:32.37459405 +0000 UTC m=+0.735677495 container attach f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T02:19:32.836 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T02:19:32.836 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 bash[75005]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate[75020]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-27T02:19:32.837 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:32 smithi008 podman[75005]: 2024-04-27 02:19:32.840796611 +0000 UTC m=+1.201880062 container died f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-27T02:19:32.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222162 10.30 MB/sec execute 399 sec latency 67.639 ms 2024-04-27T02:19:33.888 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:33 smithi008 ceph-mon[63900]: pgmap v142: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 489 op/s; 146/1596 objects degraded (9.148%) 2024-04-27T02:19:33.888 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:33 smithi008 ceph-mon[63900]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T02:19:33.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222425 10.29 MB/sec execute 400 sec latency 72.244 ms 2024-04-27T02:19:34.139 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:33 smithi008 podman[75140]: 2024-04-27 02:19:33.925673899 +0000 UTC m=+1.078840704 container remove f14dce101c606c7c31866c33e01a833b2b3f296d3a23d5dc641f609bb6c54bad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-27T02:19:34.139 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 podman[75206]: 2024-04-27 02:19:34.056592721 +0000 UTC m=+0.018956313 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:19:34.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:33 smithi077 ceph-mon[50325]: pgmap v142: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 489 op/s; 146/1596 objects degraded (9.148%) 2024-04-27T02:19:34.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:33 smithi077 ceph-mon[50325]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T02:19:34.400 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 podman[75206]: 2024-04-27 02:19:34.18550904 +0000 UTC m=+0.147872637 container create 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T02:19:34.651 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 podman[75206]: 2024-04-27 02:19:34.548188598 +0000 UTC m=+0.510552204 container init 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T02:19:34.651 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 podman[75206]: 2024-04-27 02:19:34.558733399 +0000 UTC m=+0.521096993 container start 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-27T02:19:34.651 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 ceph-osd[75256]: -- 172.21.15.8:0/839054084 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5633ecffb680 con 0x5633ecfc4c00 2024-04-27T02:19:34.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222622 10.27 MB/sec execute 401 sec latency 92.275 ms 2024-04-27T02:19:34.902 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 bash[75206]: 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 2024-04-27T02:19:34.902 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:34 smithi008 systemd[1]: Started Ceph osd.2 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:19:35.153 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:35 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:19:35.133+0000 7fb6847a67c0 -1 Falling back to public interface 2024-04-27T02:19:35.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 222906 10.28 MB/sec execute 402 sec latency 74.888 ms 2024-04-27T02:19:35.942 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:35.942 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:35.942 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:35 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:36.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:35 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:36.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223274 10.27 MB/sec execute 403 sec latency 79.403 ms 2024-04-27T02:19:36.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:36 smithi008 ceph-mon[63900]: pgmap v143: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.7 MiB/s wr, 466 op/s; 144/1578 objects degraded (9.125%) 2024-04-27T02:19:36.956 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:36 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 144/1578 objects degraded (9.125%), 19 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:36 smithi077 ceph-mon[50325]: pgmap v143: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.7 MiB/s wr, 466 op/s; 144/1578 objects degraded (9.125%) 2024-04-27T02:19:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:36 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 144/1578 objects degraded (9.125%), 19 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:37.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 223639 10.28 MB/sec execute 404 sec latency 52.476 ms 2024-04-27T02:19:38.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224023 10.28 MB/sec execute 405 sec latency 62.261 ms 2024-04-27T02:19:38.974 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:38 smithi008 ceph-mon[63900]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.7 MiB/s wr, 357 op/s; 143/1569 objects degraded (9.114%) 2024-04-27T02:19:39.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:38 smithi077 ceph-mon[50325]: pgmap v144: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 352 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.7 MiB/s wr, 357 op/s; 143/1569 objects degraded (9.114%) 2024-04-27T02:19:39.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224343 10.27 MB/sec execute 406 sec latency 63.376 ms 2024-04-27T02:19:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:19:40.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 224709 10.27 MB/sec execute 407 sec latency 45.078 ms 2024-04-27T02:19:41.002 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:40 smithi008 ceph-mon[63900]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.4 MiB/s wr, 314 op/s; 147/1596 objects degraded (9.211%) 2024-04-27T02:19:41.002 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:40 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:19:40.915+0000 7fb6847a67c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-27T02:19:41.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:40 smithi077 ceph-mon[50325]: pgmap v145: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 353 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.4 MiB/s wr, 314 op/s; 147/1596 objects degraded (9.211%) 2024-04-27T02:19:41.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225029 10.26 MB/sec execute 408 sec latency 36.386 ms 2024-04-27T02:19:42.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:41 smithi008 ceph-mon[63900]: from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:42.019 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:41 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:42.020 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:19:42.020 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 46s ago 15m 16.4M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:19:42.020 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (2m) 46s ago 15m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:19:42.020 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (2m) 2m ago 14m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 46s ago 14m - - 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (9m) 46s ago 9m 40.9M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (9m) 2m ago 9m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (5m) 46s ago 16m 505M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (3m) 46s ago 16m 51.1M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (3m) 2m ago 14m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (9m) 46s ago 9m 279M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (8m) 2m ago 8m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (5m) 46s ago 14m 10.9M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:19:42.021 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (5m) 2m ago 14m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (106s) 46s ago 13m 256M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (55s) 46s ago 13m 12.3M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 starting - - - 1937M 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 46s ago 12m 376M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (11m) 2m ago 11m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (11m) 2m ago 11m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (11m) 2m ago 11m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (10m) 2m ago 10m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:19:42.022 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (4m) 46s ago 14m 63.0M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:19:42.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:41 smithi077 ceph-mon[50325]: from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:42.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:41 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:42.270 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:19:42.220+0000 7fb6847a67c0 -1 osd.2 96 log_to_monitors true 2024-04-27T02:19:42.707 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:19:42.707 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:19:42.707 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:42.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:42.707 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 6 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:19:42.708 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:19:42.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225374 10.26 MB/sec execute 409 sec latency 72.738 ms 2024-04-27T02:19:43.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:43.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: pgmap v146: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 357 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 152/1623 objects degraded (9.365%) 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 152/1623 objects degraded (9.365%), 19 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:43.031 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:42 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/640687515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: pgmap v146: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 357 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 152/1623 objects degraded (9.365%) 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 152/1623 objects degraded (9.365%), 19 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:43.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:42 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/640687515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:19:43.339 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:19:43.340 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:19:43.340 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "9/23 daemons upgraded", 2024-04-27T02:19:43.340 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:19:43.340 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:19:43.340 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:19:43.574 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:19:43.356+0000 7fb671bf6700 -1 osd.2 96 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: osdmap e99: 8 total, 7 up, 8 in 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: from='osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:19:43.826 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:43 smithi008 ceph-mon[63900]: from='osd.2 ' entity='osd.2' 2024-04-27T02:19:43.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225630 10.25 MB/sec execute 410 sec latency 68.047 ms 2024-04-27T02:19:44.042 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 152/1623 objects degraded (9.365%), 19 pgs degraded 2024-04-27T02:19:44.042 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:19:44.042 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: osd.2 (root=default,host=smithi008) is down 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 152/1623 objects degraded (9.365%), 19 pgs degraded 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-04-27T02:19:44.043 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-04-27T02:19:44.044 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: osdmap e99: 8 total, 7 up, 8 in 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: from='osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:19:44.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:43 smithi077 ceph-mon[50325]: from='osd.2 ' entity='osd.2' 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: pgmap v148: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 467 op/s; 155/1668 objects degraded (9.293%) 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1362821463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960] boot 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:44.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:44.877 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:44 smithi008 ceph-mon[63900]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T02:19:44.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 225823 10.23 MB/sec execute 411 sec latency 567.555 ms 2024-04-27T02:19:45.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:19:45.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: pgmap v148: 97 pgs: 7 active+undersized, 19 active+undersized+degraded, 71 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 467 op/s; 155/1668 objects degraded (9.293%) 2024-04-27T02:19:45.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1362821463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:19:45.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:19:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: osd.2 [v2:172.21.15.8:6818/1961375960,v1:172.21.15.8:6819/1961375960] boot 2024-04-27T02:19:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T02:19:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T02:19:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:45.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:45.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:44 smithi077 ceph-mon[50325]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T02:19:45.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226158 10.23 MB/sec execute 412 sec latency 1184.419 ms 2024-04-27T02:19:46.697 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:46 smithi008 ceph-mon[63900]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T02:19:46.697 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:46 smithi008 ceph-mon[63900]: pgmap v152: 97 pgs: 2 remapped+peering, 5 active+undersized, 15 active+undersized+degraded, 75 active+clean; 360 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 643 op/s; 105/1635 objects degraded (6.422%); 43 KiB/s, 3 objects/s recovering 2024-04-27T02:19:46.697 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:46.697 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:46 smithi077 ceph-mon[50325]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T02:19:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:46 smithi077 ceph-mon[50325]: pgmap v152: 97 pgs: 2 remapped+peering, 5 active+undersized, 15 active+undersized+degraded, 75 active+clean; 360 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 643 op/s; 105/1635 objects degraded (6.422%); 43 KiB/s, 3 objects/s recovering 2024-04-27T02:19:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:46.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:19:46.900 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226436 10.23 MB/sec execute 413 sec latency 72.915 ms 2024-04-27T02:19:47.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226702 10.23 MB/sec execute 414 sec latency 718.678 ms 2024-04-27T02:19:48.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:48 smithi008 ceph-mon[63900]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T02:19:48.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:48 smithi008 ceph-mon[63900]: pgmap v154: 97 pgs: 2 remapped+peering, 1 active+undersized, 1 active+undersized+remapped, 4 active+undersized+degraded, 89 active+clean; 354 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 4.6 MiB/s wr, 416 op/s; 21/1560 objects degraded (1.346%); 43 KiB/s, 27 objects/s recovering 2024-04-27T02:19:48.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:48 smithi077 ceph-mon[50325]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T02:19:48.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:48 smithi077 ceph-mon[50325]: pgmap v154: 97 pgs: 2 remapped+peering, 1 active+undersized, 1 active+undersized+remapped, 4 active+undersized+degraded, 89 active+clean; 354 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 4.6 MiB/s wr, 416 op/s; 21/1560 objects degraded (1.346%); 43 KiB/s, 27 objects/s recovering 2024-04-27T02:19:48.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 226938 10.21 MB/sec execute 415 sec latency 1032.366 ms 2024-04-27T02:19:49.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:49 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 21/1560 objects degraded (1.346%), 4 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:49.236 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:49 smithi008 ceph-mon[63900]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T02:19:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:49 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 21/1560 objects degraded (1.346%), 4 pgs degraded (PG_DEGRADED) 2024-04-27T02:19:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:49 smithi077 ceph-mon[50325]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T02:19:49.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227257 10.21 MB/sec execute 416 sec latency 59.246 ms 2024-04-27T02:19:50.253 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:50 smithi008 ceph-mon[63900]: pgmap v156: 97 pgs: 1 remapped+peering, 1 active+undersized+remapped, 95 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.2 MiB/s wr, 344 op/s; 109 KiB/s, 26 objects/s recovering 2024-04-27T02:19:50.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:50 smithi077 ceph-mon[50325]: pgmap v156: 97 pgs: 1 remapped+peering, 1 active+undersized+remapped, 95 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.2 MiB/s wr, 344 op/s; 109 KiB/s, 26 objects/s recovering 2024-04-27T02:19:50.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227419 10.19 MB/sec execute 417 sec latency 47.753 ms 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1560 objects degraded (1.346%), 4 pgs degraded) 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.280 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.281 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:51 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1560 objects degraded (1.346%), 4 pgs degraded) 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:51.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:19:51.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:19:51.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:51 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:19:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 227765 10.20 MB/sec execute 418 sec latency 48.833 ms 2024-04-27T02:19:52.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:52 smithi008 ceph-mon[63900]: pgmap v157: 97 pgs: 1 active+undersized+remapped, 96 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.3 MiB/s wr, 253 op/s; 222 KiB/s, 27 objects/s recovering 2024-04-27T02:19:52.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:52 smithi008 ceph-mon[63900]: pgmap v158: 97 pgs: 1 active+undersized+remapped, 96 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.4 MiB/s wr, 267 op/s; 233 KiB/s, 29 objects/s recovering 2024-04-27T02:19:52.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:52 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:19:52.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:52 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-27T02:19:52.569 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:52.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228082 10.20 MB/sec execute 419 sec latency 55.255 ms 2024-04-27T02:19:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:52 smithi077 ceph-mon[50325]: pgmap v157: 97 pgs: 1 active+undersized+remapped, 96 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.3 MiB/s wr, 253 op/s; 222 KiB/s, 27 objects/s recovering 2024-04-27T02:19:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:52 smithi077 ceph-mon[50325]: pgmap v158: 97 pgs: 1 active+undersized+remapped, 96 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.4 MiB/s wr, 267 op/s; 233 KiB/s, 29 objects/s recovering 2024-04-27T02:19:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:52 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:19:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:52 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-27T02:19:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:19:53.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228389 10.20 MB/sec execute 420 sec latency 62.178 ms 2024-04-27T02:19:54.588 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:54 smithi008 ceph-mon[63900]: pgmap v159: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.3 MiB/s wr, 442 op/s; 210 KiB/s, 12 objects/s recovering 2024-04-27T02:19:54.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228650 10.19 MB/sec execute 421 sec latency 67.393 ms 2024-04-27T02:19:54.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:54 smithi077 ceph-mon[50325]: pgmap v159: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.3 MiB/s wr, 442 op/s; 210 KiB/s, 12 objects/s recovering 2024-04-27T02:19:55.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 228904 10.17 MB/sec execute 422 sec latency 56.985 ms 2024-04-27T02:19:56.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:56 smithi008 ceph-mon[63900]: pgmap v160: 97 pgs: 97 active+clean; 358 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 3.5 MiB/s wr, 348 op/s; 168 KiB/s, 7 objects/s recovering 2024-04-27T02:19:56.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229178 10.16 MB/sec execute 423 sec latency 61.887 ms 2024-04-27T02:19:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:56 smithi077 ceph-mon[50325]: pgmap v160: 97 pgs: 97 active+clean; 358 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 3.5 MiB/s wr, 348 op/s; 168 KiB/s, 7 objects/s recovering 2024-04-27T02:19:57.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229388 10.14 MB/sec execute 424 sec latency 54.441 ms 2024-04-27T02:19:58.671 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:19:58 smithi008 ceph-mon[63900]: pgmap v161: 97 pgs: 97 active+clean; 358 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.1 MiB/s wr, 292 op/s; 127 KiB/s, 5 objects/s recovering 2024-04-27T02:19:58.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 229758 10.15 MB/sec execute 425 sec latency 40.254 ms 2024-04-27T02:19:58.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:19:58 smithi077 ceph-mon[50325]: pgmap v161: 97 pgs: 97 active+clean; 358 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.1 MiB/s wr, 292 op/s; 127 KiB/s, 5 objects/s recovering 2024-04-27T02:19:59.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230265 10.16 MB/sec execute 426 sec latency 38.960 ms 2024-04-27T02:20:00.701 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:00 smithi008 ceph-mon[63900]: pgmap v162: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.3 MiB/s wr, 494 op/s; 87 KiB/s, 2 objects/s recovering 2024-04-27T02:20:00.701 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:00 smithi008 ceph-mon[63900]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:20:00.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230682 10.16 MB/sec execute 427 sec latency 48.295 ms 2024-04-27T02:20:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:00 smithi077 ceph-mon[50325]: pgmap v162: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.3 MiB/s wr, 494 op/s; 87 KiB/s, 2 objects/s recovering 2024-04-27T02:20:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:00 smithi077 ceph-mon[50325]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:20:01.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 230987 10.16 MB/sec execute 428 sec latency 36.238 ms 2024-04-27T02:20:02.222 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:02 smithi008 ceph-mon[63900]: pgmap v163: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 489 op/s 2024-04-27T02:20:02.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:02.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:02.223 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:02.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:02 smithi077 ceph-mon[50325]: pgmap v163: 97 pgs: 97 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 489 op/s 2024-04-27T02:20:02.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:02.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:02.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:02.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231366 10.17 MB/sec execute 429 sec latency 62.993 ms 2024-04-27T02:20:03.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231662 10.16 MB/sec execute 430 sec latency 44.197 ms 2024-04-27T02:20:04.265 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:04 smithi008 ceph-mon[63900]: pgmap v164: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 474 op/s 2024-04-27T02:20:04.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:04 smithi077 ceph-mon[50325]: pgmap v164: 97 pgs: 97 active+clean; 355 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 474 op/s 2024-04-27T02:20:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 231957 10.17 MB/sec execute 431 sec latency 42.667 ms 2024-04-27T02:20:05.901 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232278 10.15 MB/sec execute 432 sec latency 39.583 ms 2024-04-27T02:20:06.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:06 smithi008 ceph-mon[63900]: pgmap v165: 97 pgs: 97 active+clean; 356 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 347 op/s 2024-04-27T02:20:06.306 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:20:06.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:06 smithi077 ceph-mon[50325]: pgmap v165: 97 pgs: 97 active+clean; 356 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 347 op/s 2024-04-27T02:20:06.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:20:06.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232635 10.14 MB/sec execute 433 sec latency 44.239 ms 2024-04-27T02:20:07.330 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:07 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:20:07.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:07 smithi008 ceph-mon[63900]: Upgrade: osd.3 is safe to restart 2024-04-27T02:20:07.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:07 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T02:20:07.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:07 smithi077 ceph-mon[50325]: Upgrade: osd.3 is safe to restart 2024-04-27T02:20:07.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 232889 10.13 MB/sec execute 434 sec latency 62.118 ms 2024-04-27T02:20:08.219 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:08 smithi008 ceph-mon[63900]: pgmap v166: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 478 op/s 2024-04-27T02:20:08.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:08 smithi077 ceph-mon[50325]: pgmap v166: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 478 op/s 2024-04-27T02:20:08.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233195 10.13 MB/sec execute 435 sec latency 52.712 ms 2024-04-27T02:20:09.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 233635 10.14 MB/sec execute 436 sec latency 35.357 ms 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: pgmap v167: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.6 MiB/s wr, 586 op/s 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: Upgrade: Updating osd.3 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:10.366 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:10 smithi008 ceph-mon[63900]: Deploying daemon osd.3 on smithi008 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: pgmap v167: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.6 MiB/s wr, 586 op/s 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: Upgrade: Updating osd.3 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:10 smithi077 ceph-mon[50325]: Deploying daemon osd.3 on smithi008 2024-04-27T02:20:10.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234046 10.14 MB/sec execute 437 sec latency 55.799 ms 2024-04-27T02:20:11.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234404 10.14 MB/sec execute 438 sec latency 38.463 ms 2024-04-27T02:20:12.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:12 smithi008 ceph-mon[63900]: pgmap v168: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-04-27T02:20:12.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:12 smithi077 ceph-mon[50325]: pgmap v168: 97 pgs: 97 active+clean; 362 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-04-27T02:20:12.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 234791 10.14 MB/sec execute 439 sec latency 52.939 ms 2024-04-27T02:20:13.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235178 10.14 MB/sec execute 440 sec latency 59.600 ms 2024-04-27T02:20:14.214 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:14 smithi008 ceph-mon[63900]: pgmap v169: 97 pgs: 97 active+clean; 361 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-04-27T02:20:14.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:14 smithi077 ceph-mon[50325]: pgmap v169: 97 pgs: 97 active+clean; 361 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-04-27T02:20:14.645 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:20:14.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235436 10.14 MB/sec execute 441 sec latency 69.419 ms 2024-04-27T02:20:14.994 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:14 smithi008 systemd[1]: Stopping Ceph osd.3 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:20:15.209 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:15 smithi008 ceph-mon[63900]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:15.375 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:15 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[43404]: 2024-04-27T02:20:15.342+0000 7f143754e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:20:15.375 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:15 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[43404]: 2024-04-27T02:20:15.342+0000 7f143754e700 -1 osd.3 104 *** Got signal Terminated *** 2024-04-27T02:20:15.375 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:15 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[43404]: 2024-04-27T02:20:15.342+0000 7f143754e700 -1 osd.3 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:20:15.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:15 smithi077 ceph-mon[50325]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:15.792 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:20:15.792 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (6m) 33s ago 15m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:20:15.792 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 33s ago 15m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:20:15.792 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (2m) 2m ago 15m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:20:15.792 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 33s ago 15m - - 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (9m) 33s ago 9m 40.7M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (9m) 2m ago 9m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (5m) 33s ago 17m 507M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (4m) 2m ago 15m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (4m) 33s ago 17m 51.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (3m) 2m ago 14m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (9m) 33s ago 9m 285M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (9m) 2m ago 9m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (6m) 33s ago 15m 12.6M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (6m) 2m ago 14m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 33s ago 14m 314M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (89s) 33s ago 13m 225M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (41s) 33s ago 13m 15.0M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (12m) 33s ago 12m 409M 1937M 17.2.0 e1d6a67b021e d23f333c53bf 2024-04-27T02:20:15.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (12m) 2m ago 12m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:20:15.794 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (12m) 2m ago 12m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:20:15.794 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (11m) 2m ago 11m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:20:15.794 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (11m) 2m ago 11m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:20:15.794 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 33s ago 14m 63.2M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:20:15.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235714 10.14 MB/sec execute 442 sec latency 56.476 ms 2024-04-27T02:20:16.377 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:20:16.377 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 3 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:20:16.378 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T02:20:16.379 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 7 2024-04-27T02:20:16.379 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:20:16.379 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:20:16.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:16 smithi008 ceph-mon[63900]: pgmap v170: 97 pgs: 97 active+clean; 362 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 503 op/s 2024-04-27T02:20:16.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:16 smithi008 ceph-mon[63900]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:16.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:16 smithi008 ceph-mon[63900]: osd.3 marked itself down and dead 2024-04-27T02:20:16.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:16 smithi008 ceph-mon[63900]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:16.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:16.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:16 smithi077 ceph-mon[50325]: pgmap v170: 97 pgs: 97 active+clean; 362 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 503 op/s 2024-04-27T02:20:16.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:16 smithi077 ceph-mon[50325]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:16 smithi077 ceph-mon[50325]: osd.3 marked itself down and dead 2024-04-27T02:20:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:16 smithi077 ceph-mon[50325]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "9/23 daemons upgraded", 2024-04-27T02:20:16.734 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:20:16.735 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:20:16.735 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:20:16.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 235823 10.12 MB/sec execute 443 sec latency 721.923 ms 2024-04-27T02:20:17.157 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:16 smithi008 podman[77390]: 2024-04-27 02:20:16.929548158 +0000 UTC m=+1.815429867 container died d23f333c53bfc66bdc637c3594ab83214476b17894eb58407d6ed3c737708765 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, architecture=x86_64, name=centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, release=754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, distribution-scope=public, io.openshift.expose-services=, version=8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8) 2024-04-27T02:20:17.183 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down 2024-04-27T02:20:17.183 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:20:17.184 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:20:17.184 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T02:20:17.184 INFO:teuthology.orchestra.run.smithi008.stdout: osd.3 (root=default,host=smithi008) is down 2024-04-27T02:20:17.434 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:17 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:20:17.434 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:17 smithi008 ceph-mon[63900]: osdmap e105: 8 total, 7 up, 8 in 2024-04-27T02:20:17.434 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:17 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/325009830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:17.434 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:17 smithi008 ceph-mon[63900]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:17.434 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:17 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/4210858316' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:20:17.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:17 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:20:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:17 smithi077 ceph-mon[50325]: osdmap e105: 8 total, 7 up, 8 in 2024-04-27T02:20:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:17 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/325009830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:17 smithi077 ceph-mon[50325]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:17 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/4210858316' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:20:17.685 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:17 smithi008 podman[77390]: 2024-04-27 02:20:17.581363289 +0000 UTC m=+2.467244995 container cleanup d23f333c53bfc66bdc637c3594ab83214476b17894eb58407d6ed3c737708765 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, name=centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, architecture=x86_64, RELEASE=HEAD, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc.) 2024-04-27T02:20:17.685 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:17 smithi008 bash[77390]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3 2024-04-27T02:20:17.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236100 10.10 MB/sec execute 444 sec latency 1092.285 ms 2024-04-27T02:20:18.186 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:17 smithi008 podman[77498]: 2024-04-27 02:20:17.959675625 +0000 UTC m=+1.024547311 container remove d23f333c53bfc66bdc637c3594ab83214476b17894eb58407d6ed3c737708765 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.tags=base centos centos-stream, distribution-scope=public, RELEASE=HEAD, release=754) 2024-04-27T02:20:18.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:18 smithi008 ceph-mon[63900]: pgmap v172: 97 pgs: 16 stale+active+clean, 81 active+clean; 364 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 427 op/s 2024-04-27T02:20:18.439 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:18 smithi008 ceph-mon[63900]: osdmap e106: 8 total, 7 up, 8 in 2024-04-27T02:20:18.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:18 smithi077 ceph-mon[50325]: pgmap v172: 97 pgs: 16 stale+active+clean, 81 active+clean; 364 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 427 op/s 2024-04-27T02:20:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:18 smithi077 ceph-mon[50325]: osdmap e106: 8 total, 7 up, 8 in 2024-04-27T02:20:18.701 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:18 smithi008 podman[77580]: 2024-04-27 02:20:18.453239541 +0000 UTC m=+0.031043006 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:20:18.701 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:18 smithi008 podman[77580]: 2024-04-27 02:20:18.667272033 +0000 UTC m=+0.245075470 container create 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-27T02:20:18.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236434 10.09 MB/sec execute 445 sec latency 26.002 ms 2024-04-27T02:20:19.206 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:19 smithi008 podman[77580]: 2024-04-27 02:20:19.02445249 +0000 UTC m=+0.602255951 container init 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-27T02:20:19.206 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:19 smithi008 podman[77580]: 2024-04-27 02:20:19.034591603 +0000 UTC m=+0.612395048 container start 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T02:20:19.206 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:19 smithi008 podman[77580]: 2024-04-27 02:20:19.19224835 +0000 UTC m=+0.770051815 container attach 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-27T02:20:19.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:19 smithi008 ceph-mon[63900]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-04-27T02:20:19.466 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:19 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 13/1668 objects degraded (0.779%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:19.728 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:19 smithi008 podman[77580]: 2024-04-27 02:20:19.527526256 +0000 UTC m=+1.105329704 container died 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-27T02:20:19.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:19 smithi077 ceph-mon[50325]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-04-27T02:20:19.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:19 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 13/1668 objects degraded (0.779%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:19.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 236795 10.10 MB/sec execute 446 sec latency 39.116 ms 2024-04-27T02:20:20.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:20 smithi008 ceph-mon[63900]: pgmap v174: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 19 peering, 7 stale+active+clean, 67 active+clean; 364 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 572 op/s; 13/1668 objects degraded (0.779%) 2024-04-27T02:20:20.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:20 smithi077 ceph-mon[50325]: pgmap v174: 97 pgs: 1 active+undersized, 3 active+undersized+degraded, 19 peering, 7 stale+active+clean, 67 active+clean; 364 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 572 op/s; 13/1668 objects degraded (0.779%) 2024-04-27T02:20:20.754 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:20 smithi008 podman[77604]: 2024-04-27 02:20:20.577679736 +0000 UTC m=+1.034570451 container remove 9b3b88cda4e3da7b445deacbb7d62b4363319287fad618f604d8b9160a613730 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:20:20.754 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:20 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-27T02:20:20.755 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:20 smithi008 systemd[1]: Stopped Ceph osd.3 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:20:20.755 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:20 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.3.service: Consumed 4min 51.705s CPU time. 2024-04-27T02:20:20.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237272 10.10 MB/sec execute 447 sec latency 42.062 ms 2024-04-27T02:20:21.009 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 systemd[1]: Starting Ceph osd.3 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:20:21.512 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 podman[77694]: 2024-04-27 02:20:21.202065122 +0000 UTC m=+0.026594318 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:20:21.512 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 podman[77694]: 2024-04-27 02:20:21.34433944 +0000 UTC m=+0.168868633 container create 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:20:21.763 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 podman[77694]: 2024-04-27 02:20:21.753362408 +0000 UTC m=+0.577891600 container init 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_CLEAN=True) 2024-04-27T02:20:21.763 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 podman[77694]: 2024-04-27 02:20:21.761873593 +0000 UTC m=+0.586402784 container start 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T02:20:21.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237663 10.10 MB/sec execute 448 sec latency 54.048 ms 2024-04-27T02:20:22.016 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:21 smithi008 podman[77694]: 2024-04-27 02:20:21.97866715 +0000 UTC m=+0.803196353 container attach 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T02:20:22.273 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-mon[63900]: pgmap v175: 97 pgs: 6 active+undersized, 16 active+undersized+degraded, 19 peering, 56 active+clean; 358 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 MiB/s wr, 403 op/s; 109/1620 objects degraded (6.728%) 2024-04-27T02:20:22.298 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T02:20:22.298 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T02:20:22.298 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 bash[77694]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-27T02:20:22.299 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T02:20:22.300 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T02:20:22.300 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-27T02:20:22.300 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T02:20:22.300 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate[77708]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-27T02:20:22.524 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:22 smithi008 podman[77829]: 2024-04-27 02:20:22.380987003 +0000 UTC m=+0.025505969 container died 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-27T02:20:22.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:22 smithi077 ceph-mon[50325]: pgmap v175: 97 pgs: 6 active+undersized, 16 active+undersized+degraded, 19 peering, 56 active+clean; 358 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 MiB/s wr, 403 op/s; 109/1620 objects degraded (6.728%) 2024-04-27T02:20:22.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 237978 10.10 MB/sec execute 449 sec latency 39.636 ms 2024-04-27T02:20:23.563 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:23 smithi008 ceph-mon[63900]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-04-27T02:20:23.563 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:23 smithi008 podman[77829]: 2024-04-27 02:20:23.393728325 +0000 UTC m=+1.038247265 container remove 34d09de974a86642465bdc0aa513829861b238965fd1368688586fb53bd7f772 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-27T02:20:23.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:23 smithi077 ceph-mon[50325]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-04-27T02:20:23.813 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:23 smithi008 podman[77893]: 2024-04-27 02:20:23.540526578 +0000 UTC m=+0.020942118 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:20:23.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238339 10.11 MB/sec execute 450 sec latency 47.799 ms 2024-04-27T02:20:24.317 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 podman[77893]: 2024-04-27 02:20:24.094033652 +0000 UTC m=+0.574449172 container create 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T02:20:24.575 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:24 smithi008 ceph-mon[63900]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 358 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 498 op/s; 209/1677 objects degraded (12.463%) 2024-04-27T02:20:24.575 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 podman[77893]: 2024-04-27 02:20:24.548718069 +0000 UTC m=+1.029133594 container init 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-27T02:20:24.575 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 podman[77893]: 2024-04-27 02:20:24.554368809 +0000 UTC m=+1.034784330 container start 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0) 2024-04-27T02:20:24.575 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 ceph-osd[77912]: -- 172.21.15.8:0/100717661 <== mon.0 v2:172.21.15.8:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602f3c25680 con 0x5602f3bef000 2024-04-27T02:20:24.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:24 smithi077 ceph-mon[50325]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 358 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.0 MiB/s wr, 498 op/s; 209/1677 objects degraded (12.463%) 2024-04-27T02:20:24.825 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 bash[77893]: 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 2024-04-27T02:20:24.826 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:24 smithi008 systemd[1]: Started Ceph osd.3 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:20:24.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238703 10.11 MB/sec execute 451 sec latency 70.724 ms 2024-04-27T02:20:25.330 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:25 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:20:25.118+0000 7f435a50a7c0 -1 Falling back to public interface 2024-04-27T02:20:25.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:25 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 209/1677 objects degraded (12.463%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:25.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:25.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:25.331 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:20:25.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:25 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 209/1677 objects degraded (12.463%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:25.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:25.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:25.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:20:25.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 238995 10.11 MB/sec execute 452 sec latency 41.115 ms 2024-04-27T02:20:26.871 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:26 smithi008 ceph-mon[63900]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 359 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 470 op/s; 202/1647 objects degraded (12.265%) 2024-04-27T02:20:26.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239300 10.10 MB/sec execute 453 sec latency 43.863 ms 2024-04-27T02:20:27.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:26 smithi077 ceph-mon[50325]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 359 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 470 op/s; 202/1647 objects degraded (12.265%) 2024-04-27T02:20:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239665 10.09 MB/sec execute 454 sec latency 73.482 ms 2024-04-27T02:20:28.902 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:28 smithi008 ceph-mon[63900]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 362 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 482 op/s; 210/1671 objects degraded (12.567%) 2024-04-27T02:20:28.902 INFO:teuthology.orchestra.run.smithi008.stdout: 5 239873 10.07 MB/sec execute 455 sec latency 81.120 ms 2024-04-27T02:20:29.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:28 smithi077 ceph-mon[50325]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 362 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 482 op/s; 210/1671 objects degraded (12.567%) 2024-04-27T02:20:29.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240043 10.06 MB/sec execute 456 sec latency 58.565 ms 2024-04-27T02:20:29.947 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:29 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 210/1671 objects degraded (12.567%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:30.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:29 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 210/1671 objects degraded (12.567%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:30.462 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:30 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:20:30.234+0000 7f435a50a7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-27T02:20:30.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240319 10.06 MB/sec execute 457 sec latency 82.125 ms 2024-04-27T02:20:31.020 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:30 smithi008 ceph-mon[63900]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 356 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.3 MiB/s wr, 432 op/s; 212/1641 objects degraded (12.919%) 2024-04-27T02:20:31.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:30 smithi077 ceph-mon[50325]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 356 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.3 MiB/s wr, 432 op/s; 212/1641 objects degraded (12.919%) 2024-04-27T02:20:31.779 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:31 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:20:31.666+0000 7f435a50a7c0 -1 osd.3 104 log_to_monitors true 2024-04-27T02:20:31.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240630 10.06 MB/sec execute 458 sec latency 42.399 ms 2024-04-27T02:20:32.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:32 smithi008 ceph-mon[63900]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 357 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.7 MiB/s wr, 460 op/s; 205/1614 objects degraded (12.701%) 2024-04-27T02:20:32.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:32.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:32.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:32 smithi008 ceph-mon[63900]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:20:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:32 smithi077 ceph-mon[50325]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 357 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.7 MiB/s wr, 460 op/s; 205/1614 objects degraded (12.701%) 2024-04-27T02:20:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:32.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:32 smithi077 ceph-mon[50325]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T02:20:32.543 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:20:32 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:20:32.553+0000 7f434795a700 -1 osd.3 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:20:32.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 240902 10.05 MB/sec execute 459 sec latency 76.796 ms 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: osdmap e107: 8 total, 7 up, 8 in 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:33.305 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:33 smithi008 ceph-mon[63900]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: osdmap e107: 8 total, 7 up, 8 in 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:33.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:33 smithi077 ceph-mon[50325]: from='osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808]' entity='osd.3' 2024-04-27T02:20:33.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241170 10.05 MB/sec execute 460 sec latency 412.301 ms 2024-04-27T02:20:34.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 513 op/s; 196/1515 objects degraded (12.937%) 2024-04-27T02:20:34.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:20:34.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808] boot 2024-04-27T02:20:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: osdmap e108: 8 total, 8 up, 8 in 2024-04-27T02:20:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:20:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:34.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:34.648 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 513 op/s; 196/1515 objects degraded (12.937%) 2024-04-27T02:20:34.648 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:20:34.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: osd.3 [v2:172.21.15.8:6826/337161808,v1:172.21.15.8:6827/337161808] boot 2024-04-27T02:20:34.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: osdmap e108: 8 total, 8 up, 8 in 2024-04-27T02:20:34.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T02:20:34.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:34.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:34.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241170 10.02 MB/sec execute 461 sec latency 1412.355 ms 2024-04-27T02:20:35.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:35 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 196/1515 objects degraded (12.937%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:35.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:35 smithi008 ceph-mon[63900]: osdmap e109: 8 total, 8 up, 8 in 2024-04-27T02:20:35.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:35 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 196/1515 objects degraded (12.937%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:35.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:35 smithi077 ceph-mon[50325]: osdmap e109: 8 total, 8 up, 8 in 2024-04-27T02:20:35.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241238 10.00 MB/sec execute 462 sec latency 2159.305 ms 2024-04-27T02:20:36.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:36 smithi008 ceph-mon[63900]: pgmap v185: 97 pgs: 4 remapped+peering, 1 active+recovery_wait+degraded, 12 active+undersized, 21 active+undersized+degraded, 59 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.1 MiB/s wr, 487 op/s; 142/1518 objects degraded (9.354%) 2024-04-27T02:20:36.747 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:36 smithi008 ceph-mon[63900]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T02:20:36.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241369 9.99 MB/sec execute 463 sec latency 424.068 ms 2024-04-27T02:20:36.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:36 smithi077 ceph-mon[50325]: pgmap v185: 97 pgs: 4 remapped+peering, 1 active+recovery_wait+degraded, 12 active+undersized, 21 active+undersized+degraded, 59 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.1 MiB/s wr, 487 op/s; 142/1518 objects degraded (9.354%) 2024-04-27T02:20:36.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:36 smithi077 ceph-mon[50325]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T02:20:37.746 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:37 smithi008 ceph-mon[63900]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T02:20:37.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241434 9.97 MB/sec execute 464 sec latency 1047.854 ms 2024-04-27T02:20:37.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:37 smithi077 ceph-mon[50325]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T02:20:38.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:38 smithi008 ceph-mon[63900]: pgmap v188: 97 pgs: 6 remapped+peering, 1 active+recovery_wait+degraded, 11 active+undersized, 17 active+undersized+degraded, 62 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s wr, 110 op/s; 124/1551 objects degraded (7.995%); 38 KiB/s, 5 objects/s recovering 2024-04-27T02:20:38.756 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:38 smithi008 ceph-mon[63900]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T02:20:38.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 241738 9.97 MB/sec execute 465 sec latency 963.904 ms 2024-04-27T02:20:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:38 smithi077 ceph-mon[50325]: pgmap v188: 97 pgs: 6 remapped+peering, 1 active+recovery_wait+degraded, 11 active+undersized, 17 active+undersized+degraded, 62 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 MiB/s wr, 110 op/s; 124/1551 objects degraded (7.995%); 38 KiB/s, 5 objects/s recovering 2024-04-27T02:20:38.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:38 smithi077 ceph-mon[50325]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.512 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:39.513 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:39 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 124/1551 objects degraded (7.995%), 18 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:39.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242058 9.97 MB/sec execute 466 sec latency 62.902 ms 2024-04-27T02:20:39.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:39.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:39 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 124/1551 objects degraded (7.995%), 18 pgs degraded (PG_DEGRADED) 2024-04-27T02:20:40.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:40 smithi008 ceph-mon[63900]: pgmap v191: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 166 op/s; 17/1572 objects degraded (1.081%); 514 KiB/s, 39 objects/s recovering 2024-04-27T02:20:40.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:40 smithi008 ceph-mon[63900]: pgmap v192: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.3 MiB/s wr, 107 op/s; 17/1572 objects degraded (1.081%); 530 KiB/s, 37 objects/s recovering 2024-04-27T02:20:40.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:40 smithi008 ceph-mon[63900]: pgmap v193: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 1.9 MiB/s wr, 149 op/s; 17/1572 objects degraded (1.081%); 736 KiB/s, 52 objects/s recovering 2024-04-27T02:20:40.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:40 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:40.525 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:40 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-27T02:20:40.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242456 9.98 MB/sec execute 467 sec latency 51.157 ms 2024-04-27T02:20:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:40 smithi077 ceph-mon[50325]: pgmap v191: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 166 op/s; 17/1572 objects degraded (1.081%); 514 KiB/s, 39 objects/s recovering 2024-04-27T02:20:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:40 smithi077 ceph-mon[50325]: pgmap v192: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.3 MiB/s wr, 107 op/s; 17/1572 objects degraded (1.081%); 530 KiB/s, 37 objects/s recovering 2024-04-27T02:20:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:40 smithi077 ceph-mon[50325]: pgmap v193: 97 pgs: 2 peering, 6 remapped+peering, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 87 active+clean; 352 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 1.9 MiB/s wr, 149 op/s; 17/1572 objects degraded (1.081%); 736 KiB/s, 52 objects/s recovering 2024-04-27T02:20:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:40 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:40 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-27T02:20:41.790 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:41 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/1572 objects degraded (1.081%), 1 pg degraded) 2024-04-27T02:20:41.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 242837 9.98 MB/sec execute 468 sec latency 47.565 ms 2024-04-27T02:20:41.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:41 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/1572 objects degraded (1.081%), 1 pg degraded) 2024-04-27T02:20:42.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:42 smithi008 ceph-mon[63900]: pgmap v194: 97 pgs: 2 peering, 2 remapped+peering, 1 active+undersized+remapped, 92 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.8 MiB/s wr, 304 op/s; 1.9 MiB/s, 47 objects/s recovering 2024-04-27T02:20:42.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243217 9.97 MB/sec execute 469 sec latency 40.955 ms 2024-04-27T02:20:42.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:42 smithi077 ceph-mon[50325]: pgmap v194: 97 pgs: 2 peering, 2 remapped+peering, 1 active+undersized+remapped, 92 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.8 MiB/s wr, 304 op/s; 1.9 MiB/s, 47 objects/s recovering 2024-04-27T02:20:43.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243465 9.96 MB/sec execute 470 sec latency 43.421 ms 2024-04-27T02:20:44.582 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:44 smithi008 ceph-mon[63900]: pgmap v195: 97 pgs: 97 active+clean; 359 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 450 op/s; 2.3 MiB/s, 20 objects/s recovering 2024-04-27T02:20:44.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 243785 9.96 MB/sec execute 471 sec latency 52.959 ms 2024-04-27T02:20:44.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:44 smithi077 ceph-mon[50325]: pgmap v195: 97 pgs: 97 active+clean; 359 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 450 op/s; 2.3 MiB/s, 20 objects/s recovering 2024-04-27T02:20:45.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244151 9.97 MB/sec execute 472 sec latency 63.593 ms 2024-04-27T02:20:46.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:46 smithi008 ceph-mon[63900]: pgmap v196: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.5 MiB/s wr, 428 op/s; 1.7 MiB/s, 17 objects/s recovering 2024-04-27T02:20:46.623 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:46.623 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:46.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244567 9.97 MB/sec execute 473 sec latency 40.390 ms 2024-04-27T02:20:46.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:46 smithi077 ceph-mon[50325]: pgmap v196: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.5 MiB/s wr, 428 op/s; 1.7 MiB/s, 17 objects/s recovering 2024-04-27T02:20:46.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:46.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:20:47.718 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:20:47.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 244833 9.96 MB/sec execute 474 sec latency 57.459 ms 2024-04-27T02:20:48.669 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:48 smithi008 ceph-mon[63900]: pgmap v197: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 1.4 MiB/s, 13 objects/s recovering 2024-04-27T02:20:48.670 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:48 smithi008 ceph-mon[63900]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7m) 16s ago 16m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (3m) 16s ago 16m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (3m) 3m ago 15m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 16s ago 16m - - 2024-04-27T02:20:48.756 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (10m) 16s ago 10m 41.1M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (10m) 3m ago 10m 14.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (6m) 16s ago 17m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (5m) 3m ago 15m 449M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 16s ago 17m 53.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (4m) 3m ago 15m 34.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (10m) 16s ago 10m 286M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (10m) 3m ago 10m 88.2M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 16s ago 16m 9675k - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (6m) 3m ago 15m 17.6M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (2m) 16s ago 14m 366M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 16s ago 14m 292M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (74s) 16s ago 13m 222M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (25s) 16s ago 13m 15.1M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (13m) 3m ago 13m 552M 2704M 17.2.0 e1d6a67b021e 748b431ef475 2024-04-27T02:20:48.757 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (12m) 3m ago 12m 384M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:20:48.758 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (12m) 3m ago 12m 568M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:20:48.758 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (11m) 3m ago 11m 457M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:20:48.758 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (5m) 16s ago 15m 63.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:20:48.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245073 9.95 MB/sec execute 475 sec latency 43.879 ms 2024-04-27T02:20:48.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:48 smithi077 ceph-mon[50325]: pgmap v197: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 1.4 MiB/s, 13 objects/s recovering 2024-04-27T02:20:48.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:48 smithi077 ceph-mon[50325]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T02:20:49.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:20:49.383 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:20:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:49 smithi008 ceph-mon[63900]: from='client.44241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:49.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:49 smithi008 ceph-mon[63900]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:49.701 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:49 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3408685552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:49.826 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:20:49.826 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "10/23 daemons upgraded", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:20:49.827 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:20:49.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245341 9.95 MB/sec execute 476 sec latency 49.500 ms 2024-04-27T02:20:49.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:49 smithi077 ceph-mon[50325]: from='client.44241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:49.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:49 smithi077 ceph-mon[50325]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:49.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:49 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3408685552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:20:50.329 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:20:50.330 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:20:50.330 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:20:50.735 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:50 smithi008 ceph-mon[63900]: pgmap v198: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 514 op/s; 1.1 MiB/s, 11 objects/s recovering 2024-04-27T02:20:50.736 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:50 smithi008 ceph-mon[63900]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:50.736 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:50 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2084329605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:20:50.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245657 9.94 MB/sec execute 477 sec latency 50.949 ms 2024-04-27T02:20:50.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:50 smithi077 ceph-mon[50325]: pgmap v198: 97 pgs: 97 active+clean; 358 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 514 op/s; 1.1 MiB/s, 11 objects/s recovering 2024-04-27T02:20:50.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:50 smithi077 ceph-mon[50325]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:20:50.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:50 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2084329605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:20:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 245939 9.95 MB/sec execute 478 sec latency 56.073 ms 2024-04-27T02:20:52.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:52 smithi008 ceph-mon[63900]: pgmap v199: 97 pgs: 97 active+clean; 354 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 448 op/s; 952 KiB/s, 9 objects/s recovering 2024-04-27T02:20:52.534 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:52.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246192 9.95 MB/sec execute 479 sec latency 53.316 ms 2024-04-27T02:20:52.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:52 smithi077 ceph-mon[50325]: pgmap v199: 97 pgs: 97 active+clean; 354 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.5 MiB/s wr, 448 op/s; 952 KiB/s, 9 objects/s recovering 2024-04-27T02:20:52.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:53.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246463 9.94 MB/sec execute 480 sec latency 93.757 ms 2024-04-27T02:20:54.520 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:54 smithi008 ceph-mon[63900]: pgmap v200: 97 pgs: 97 active+clean; 355 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 512 op/s; 417 KiB/s, 4 objects/s recovering 2024-04-27T02:20:54.520 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:54.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246769 9.92 MB/sec execute 481 sec latency 55.610 ms 2024-04-27T02:20:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:54 smithi077 ceph-mon[50325]: pgmap v200: 97 pgs: 97 active+clean; 355 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 512 op/s; 417 KiB/s, 4 objects/s recovering 2024-04-27T02:20:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:55.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:55 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:55.708 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:55 smithi008 ceph-mon[63900]: Upgrade: osd.4 is safe to restart 2024-04-27T02:20:55.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 246981 9.91 MB/sec execute 482 sec latency 64.038 ms 2024-04-27T02:20:55.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:55 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T02:20:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:55 smithi077 ceph-mon[50325]: Upgrade: osd.4 is safe to restart 2024-04-27T02:20:56.730 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:56 smithi008 ceph-mon[63900]: pgmap v201: 97 pgs: 97 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 397 op/s 2024-04-27T02:20:56.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:56.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:20:56.731 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:56.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:56 smithi077 ceph-mon[50325]: pgmap v201: 97 pgs: 97 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 397 op/s 2024-04-27T02:20:56.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:20:56.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T02:20:56.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:20:56.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247287 9.91 MB/sec execute 483 sec latency 44.782 ms 2024-04-27T02:20:57.752 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:57 smithi008 ceph-mon[63900]: Upgrade: Updating osd.4 2024-04-27T02:20:57.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:57 smithi008 ceph-mon[63900]: Deploying daemon osd.4 on smithi077 2024-04-27T02:20:57.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 247637 9.91 MB/sec execute 484 sec latency 43.806 ms 2024-04-27T02:20:57.917 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:57 smithi077 ceph-mon[50325]: Upgrade: Updating osd.4 2024-04-27T02:20:57.918 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:57 smithi077 ceph-mon[50325]: Deploying daemon osd.4 on smithi077 2024-04-27T02:20:58.544 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:20:58 smithi008 ceph-mon[63900]: pgmap v202: 97 pgs: 97 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 372 op/s 2024-04-27T02:20:58.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248031 9.92 MB/sec execute 485 sec latency 52.054 ms 2024-04-27T02:20:58.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:20:58 smithi077 ceph-mon[50325]: pgmap v202: 97 pgs: 97 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.8 MiB/s wr, 372 op/s 2024-04-27T02:20:59.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248321 9.91 MB/sec execute 486 sec latency 61.573 ms 2024-04-27T02:21:00.066 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:20:59 smithi077 systemd[1]: Stopping Ceph osd.4 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:21:00.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[28459]: 2024-04-27T02:21:00.127+0000 7f6c3aa09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:21:00.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[28459]: 2024-04-27T02:21:00.127+0000 7f6c3aa09700 -1 osd.4 113 *** Got signal Terminated *** 2024-04-27T02:21:00.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[28459]: 2024-04-27T02:21:00.127+0000 7f6c3aa09700 -1 osd.4 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:21:00.577 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:00 smithi008 ceph-mon[63900]: pgmap v203: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.1 MiB/s wr, 530 op/s 2024-04-27T02:21:00.577 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:00 smithi008 ceph-mon[63900]: osd.4 marked itself down and dead 2024-04-27T02:21:00.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248498 9.89 MB/sec execute 487 sec latency 371.243 ms 2024-04-27T02:21:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:00 smithi077 ceph-mon[50325]: pgmap v203: 97 pgs: 97 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.1 MiB/s wr, 530 op/s 2024-04-27T02:21:00.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:00 smithi077 ceph-mon[50325]: osd.4 marked itself down and dead 2024-04-27T02:21:01.450 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:01 smithi077 podman[54798]: 2024-04-27 02:21:01.194419783 +0000 UTC m=+1.205428505 container died 748b431ef4753c721f4bf8f0181455c75b7ea150b0cf90b8dffc532f75a4d643 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, release=754, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:21:01.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:01 smithi077 podman[54798]: 2024-04-27 02:21:01.565896942 +0000 UTC m=+1.576905668 container cleanup 748b431ef4753c721f4bf8f0181455c75b7ea150b0cf90b8dffc532f75a4d643 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:21:01.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:01 smithi077 bash[54798]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4 2024-04-27T02:21:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:21:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: osdmap e114: 8 total, 7 up, 8 in 2024-04-27T02:21:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5, 3, 4]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 4]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 0]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 4]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 2]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:01.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5, 3, 4]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 4]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 0]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 4]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 2]}]': finished 2024-04-27T02:21:01.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: osdmap e114: 8 total, 7 up, 8 in 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5, 3, 4]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 4]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 0]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 4]}]: dispatch 2024-04-27T02:21:01.796 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]: dispatch 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 2]}]: dispatch 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5, 3, 4]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 4]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 0]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 0]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 4]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 5]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [3, 2]}]': finished 2024-04-27T02:21:01.797 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-04-27T02:21:01.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248599 9.88 MB/sec execute 488 sec latency 1035.824 ms 2024-04-27T02:21:02.199 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:01 smithi077 podman[54824]: 2024-04-27 02:21:01.858761078 +0000 UTC m=+0.665339951 container remove 748b431ef4753c721f4bf8f0181455c75b7ea150b0cf90b8dffc532f75a4d643 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-27T02:21:02.473 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.126764748 +0000 UTC m=+0.018830717 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:02.473 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.259820239 +0000 UTC m=+0.151886203 container create a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb) 2024-04-27T02:21:02.473 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.496657852 +0000 UTC m=+0.388723822 container init a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T02:21:02.558 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:02 smithi008 ceph-mon[63900]: pgmap v205: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 410 op/s 2024-04-27T02:21:02.559 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:02 smithi008 ceph-mon[63900]: osdmap e115: 8 total, 7 up, 8 in 2024-04-27T02:21:02.559 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:02 smithi008 ceph-mon[63900]: osdmap e116: 8 total, 7 up, 8 in 2024-04-27T02:21:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:02 smithi077 ceph-mon[50325]: pgmap v205: 97 pgs: 14 stale+active+clean, 83 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.4 MiB/s wr, 410 op/s 2024-04-27T02:21:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:02 smithi077 ceph-mon[50325]: osdmap e115: 8 total, 7 up, 8 in 2024-04-27T02:21:02.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:02 smithi077 ceph-mon[50325]: osdmap e116: 8 total, 7 up, 8 in 2024-04-27T02:21:02.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.501402696 +0000 UTC m=+0.393468668 container start a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-27T02:21:02.734 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.629811047 +0000 UTC m=+0.521877014 container attach a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:21:02.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 248889 9.88 MB/sec execute 489 sec latency 58.539 ms 2024-04-27T02:21:03.142 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:02 smithi077 podman[54912]: 2024-04-27 02:21:02.813781373 +0000 UTC m=+0.705847329 container died a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:21:03.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 podman[54939]: 2024-04-27 02:21:03.417892907 +0000 UTC m=+0.593748510 container remove a665b3c1dd37b9297c7375e2edc377efb56670d91155f5d2911bca8888829d3b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-b22e2eb) 2024-04-27T02:21:03.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-27T02:21:03.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 systemd[1]: Stopped Ceph osd.4 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:21:03.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.4.service: Consumed 4min 33.381s CPU time. 2024-04-27T02:21:03.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 systemd[1]: Starting Ceph osd.4 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:21:03.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249212 9.88 MB/sec execute 490 sec latency 49.271 ms 2024-04-27T02:21:04.115 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 podman[55041]: 2024-04-27 02:21:03.783542609 +0000 UTC m=+0.022786796 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:04.115 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:03 smithi077 podman[55041]: 2024-04-27 02:21:03.894008048 +0000 UTC m=+0.133252222 container create 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:21:04.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 podman[55041]: 2024-04-27 02:21:04.138507205 +0000 UTC m=+0.377751380 container init 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:21:04.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 podman[55041]: 2024-04-27 02:21:04.142305371 +0000 UTC m=+0.381549559 container start 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb) 2024-04-27T02:21:04.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 podman[55041]: 2024-04-27 02:21:04.230227806 +0000 UTC m=+0.469471989 container attach 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T02:21:04.644 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:04 smithi008 ceph-mon[63900]: pgmap v208: 97 pgs: 3 remapped+peering, 4 peering, 5 active+undersized, 17 active+undersized+degraded, 6 stale+active+clean, 62 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 672 op/s; 148/1581 objects degraded (9.361%) 2024-04-27T02:21:04.644 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:04 smithi008 ceph-mon[63900]: osdmap e117: 8 total, 7 up, 8 in 2024-04-27T02:21:04.644 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:04 smithi008 ceph-mon[63900]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:21:04.644 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:04 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 148/1581 objects degraded (9.361%), 17 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:04.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-mon[50325]: pgmap v208: 97 pgs: 3 remapped+peering, 4 peering, 5 active+undersized, 17 active+undersized+degraded, 6 stale+active+clean, 62 active+clean; 353 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 672 op/s; 148/1581 objects degraded (9.361%) 2024-04-27T02:21:04.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-mon[50325]: osdmap e117: 8 total, 7 up, 8 in 2024-04-27T02:21:04.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-mon[50325]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-27T02:21:04.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 148/1581 objects degraded (9.361%), 17 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-27T02:21:04.902 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 bash[55041]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate[55056]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-27T02:21:04.903 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:04 smithi077 podman[55041]: 2024-04-27 02:21:04.626128756 +0000 UTC m=+0.865372924 container died 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:21:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249458 9.88 MB/sec execute 491 sec latency 59.477 ms 2024-04-27T02:21:05.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 podman[55188]: 2024-04-27 02:21:05.252106177 +0000 UTC m=+0.618082840 container remove 32132d10fc0e279383415f5c02be5264c04a89a20e15b3e448597fbc3ca08e31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-activate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T02:21:05.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 podman[55252]: 2024-04-27 02:21:05.36854799 +0000 UTC m=+0.018030163 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:05.483 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 podman[55252]: 2024-04-27 02:21:05.494663432 +0000 UTC m=+0.144145606 container create 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T02:21:05.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249709 9.87 MB/sec execute 492 sec latency 66.138 ms 2024-04-27T02:21:05.983 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 podman[55252]: 2024-04-27 02:21:05.722305933 +0000 UTC m=+0.371788125 container init 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb) 2024-04-27T02:21:05.983 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 podman[55252]: 2024-04-27 02:21:05.725318827 +0000 UTC m=+0.374801003 container start 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-27T02:21:05.983 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 ceph-osd[55271]: -- 172.21.15.77:0/1469165215 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f6fc95680 con 0x558f6fc5f000 2024-04-27T02:21:05.983 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 bash[55252]: 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 2024-04-27T02:21:05.983 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:05 smithi077 systemd[1]: Started Ceph osd.4 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:21:06.733 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:21:06.269+0000 7f9c1022b7c0 -1 Falling back to public interface 2024-04-27T02:21:06.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 249982 9.87 MB/sec execute 493 sec latency 61.143 ms 2024-04-27T02:21:06.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:06 smithi008 ceph-mon[63900]: pgmap v210: 97 pgs: 2 active+clean+remapped, 3 remapped+peering, 4 peering, 7 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 52 active+clean; 354 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 452 op/s; 245/1581 objects degraded (15.497%) 2024-04-27T02:21:06.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:06.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:06.933 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:07.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:06 smithi077 ceph-mon[50325]: pgmap v210: 97 pgs: 2 active+clean+remapped, 3 remapped+peering, 4 peering, 7 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 52 active+clean; 354 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 452 op/s; 245/1581 objects degraded (15.497%) 2024-04-27T02:21:07.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:07.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:07.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:07.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250195 9.85 MB/sec execute 494 sec latency 60.677 ms 2024-04-27T02:21:07.931 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:07 smithi077 ceph-mon[50325]: pgmap v211: 97 pgs: 2 active+clean+remapped, 3 remapped+peering, 4 peering, 7 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 52 active+clean; 354 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.9 MiB/s wr, 342 op/s; 245/1581 objects degraded (15.497%) 2024-04-27T02:21:07.950 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:07 smithi008 ceph-mon[63900]: pgmap v211: 97 pgs: 2 active+clean+remapped, 3 remapped+peering, 4 peering, 7 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 52 active+clean; 354 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.9 MiB/s wr, 342 op/s; 245/1581 objects degraded (15.497%) 2024-04-27T02:21:08.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250418 9.84 MB/sec execute 495 sec latency 53.101 ms 2024-04-27T02:21:09.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:09 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 261/1629 objects degraded (16.022%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:09.242 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:09 smithi008 ceph-mon[63900]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-27T02:21:09.481 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:09 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 261/1629 objects degraded (16.022%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:09.481 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:09 smithi077 ceph-mon[50325]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-27T02:21:09.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250630 9.83 MB/sec execute 496 sec latency 69.833 ms 2024-04-27T02:21:10.249 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:10 smithi008 ceph-mon[63900]: pgmap v212: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 356 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 501 op/s; 261/1629 objects degraded (16.022%) 2024-04-27T02:21:10.250 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:10.250 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:10.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:10 smithi077 ceph-mon[50325]: pgmap v212: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 356 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 501 op/s; 261/1629 objects degraded (16.022%) 2024-04-27T02:21:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:10.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:10.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 250947 9.83 MB/sec execute 497 sec latency 47.053 ms 2024-04-27T02:21:11.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251386 9.84 MB/sec execute 498 sec latency 48.923 ms 2024-04-27T02:21:12.286 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:12 smithi008 ceph-mon[63900]: pgmap v213: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.2 MiB/s wr, 288 op/s; 260/1632 objects degraded (15.931%) 2024-04-27T02:21:12.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:12.287 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:12.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:12 smithi077 ceph-mon[50325]: pgmap v213: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 356 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.2 MiB/s wr, 288 op/s; 260/1632 objects degraded (15.931%) 2024-04-27T02:21:12.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:12 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:12.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:12 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:12.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 251915 9.84 MB/sec execute 499 sec latency 32.556 ms 2024-04-27T02:21:13.035 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:21:12.797+0000 7f9c1022b7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-27T02:21:13.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252245 9.84 MB/sec execute 500 sec latency 42.099 ms 2024-04-27T02:21:14.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:14 smithi077 ceph-mon[50325]: pgmap v214: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 354 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 401 op/s; 246/1611 objects degraded (15.270%) 2024-04-27T02:21:14.295 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:21:14.199+0000 7f9c1022b7c0 -1 osd.4 113 log_to_monitors true 2024-04-27T02:21:14.335 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:14 smithi008 ceph-mon[63900]: pgmap v214: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 354 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 401 op/s; 246/1611 objects degraded (15.270%) 2024-04-27T02:21:14.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252550 9.84 MB/sec execute 501 sec latency 66.992 ms 2024-04-27T02:21:15.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:15 smithi008 ceph-mon[63900]: from='osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:21:15.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:15 smithi008 ceph-mon[63900]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:21:15.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:15 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 246/1611 objects degraded (15.270%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:15.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:15 smithi077 ceph-mon[50325]: from='osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:21:15.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:15 smithi077 ceph-mon[50325]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T02:21:15.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:15 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 246/1611 objects degraded (15.270%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:15.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 252783 9.83 MB/sec execute 502 sec latency 149.404 ms 2024-04-27T02:21:15.982 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:21:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:21:15.633+0000 7f9bfd67b700 -1 osd.4 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: pgmap v215: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 425 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: osdmap e118: 8 total, 7 up, 8 in 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:16.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='osd.4 ' entity='osd.4' 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: pgmap v217: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 548 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: pgmap v218: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 452 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:16.175 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:16.176 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:16 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: pgmap v215: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.3 MiB/s wr, 425 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: osdmap e118: 8 total, 7 up, 8 in 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='osd.4 ' entity='osd.4' 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: pgmap v217: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 548 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: pgmap v218: 97 pgs: 9 active+clean+remapped, 7 active+undersized, 28 active+undersized+degraded, 53 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 452 op/s; 249/1620 objects degraded (15.370%) 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:16.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:16 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-27T02:21:16.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253086 9.84 MB/sec execute 503 sec latency 277.526 ms 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441] boot 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T02:21:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:21:17.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:17.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:17.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:21:17.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: osd.4 [v2:172.21.15.77:6800/3652874441,v1:172.21.15.77:6801/3652874441] boot 2024-04-27T02:21:17.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T02:21:17.622 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T02:21:17.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253142 9.82 MB/sec execute 504 sec latency 1084.231 ms 2024-04-27T02:21:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:18 smithi008 ceph-mon[63900]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T02:21:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:18 smithi008 ceph-mon[63900]: pgmap v221: 97 pgs: 4 remapped+peering, 15 peering, 5 active+clean+remapped, 7 active+undersized, 13 active+undersized+degraded, 53 active+clean; 357 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 334 op/s; 124/1620 objects degraded (7.654%) 2024-04-27T02:21:18.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253459 9.82 MB/sec execute 505 sec latency 928.964 ms 2024-04-27T02:21:18.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:18 smithi077 ceph-mon[50325]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T02:21:18.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:18 smithi077 ceph-mon[50325]: pgmap v221: 97 pgs: 4 remapped+peering, 15 peering, 5 active+clean+remapped, 7 active+undersized, 13 active+undersized+degraded, 53 active+clean; 357 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 334 op/s; 124/1620 objects degraded (7.654%) 2024-04-27T02:21:19.714 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:19 smithi008 ceph-mon[63900]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T02:21:19.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:19 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 124/1620 objects degraded (7.654%), 13 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:19.715 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:19 smithi008 ceph-mon[63900]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T02:21:19.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253695 9.81 MB/sec execute 506 sec latency 51.192 ms 2024-04-27T02:21:19.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:19 smithi077 ceph-mon[50325]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T02:21:19.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:19 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 124/1620 objects degraded (7.654%), 13 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:19.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:19 smithi077 ceph-mon[50325]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T02:21:20.739 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:20 smithi008 ceph-mon[63900]: pgmap v224: 97 pgs: 4 remapped+peering, 17 peering, 2 active+clean+remapped, 6 active+undersized, 8 active+undersized+degraded, 60 active+clean; 356 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.1 MiB/s wr, 246 op/s; 69/1602 objects degraded (4.307%); 301 KiB/s, 12 objects/s recovering 2024-04-27T02:21:20.739 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:20 smithi008 ceph-mon[63900]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T02:21:20.884 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:21:20.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 253964 9.80 MB/sec execute 507 sec latency 258.252 ms 2024-04-27T02:21:20.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:20 smithi077 ceph-mon[50325]: pgmap v224: 97 pgs: 4 remapped+peering, 17 peering, 2 active+clean+remapped, 6 active+undersized, 8 active+undersized+degraded, 60 active+clean; 356 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.1 MiB/s wr, 246 op/s; 69/1602 objects degraded (4.307%); 301 KiB/s, 12 objects/s recovering 2024-04-27T02:21:20.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:20 smithi077 ceph-mon[50325]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T02:21:21.896 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:21:21.896 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 49s ago 16m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:21:21.896 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 49s ago 16m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (4m) 12s ago 16m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 49s ago 16m - - 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (10m) 49s ago 10m 41.1M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (10m) 12s ago 10m 15.0M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (6m) 49s ago 18m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (5m) 12s ago 16m 451M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (5m) 49s ago 18m 53.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (5m) 12s ago 16m 40.8M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (10m) 49s ago 10m 286M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (10m) 12s ago 10m 88.8M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (7m) 49s ago 16m 9675k - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:21:21.897 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (7m) 12s ago 15m 20.5M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (3m) 49s ago 15m 366M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (2m) 49s ago 14m 292M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (107s) 49s ago 14m 222M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (58s) 49s ago 13m 15.1M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (16s) 12s ago 13m 12.2M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (13m) 12s ago 13m 575M 2704M 17.2.0 e1d6a67b021e cdddbccab76c 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (12m) 12s ago 12m 513M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (12m) 12s ago 12m 796M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:21:21.898 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 49s ago 15m 63.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:21:21.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254194 9.79 MB/sec execute 508 sec latency 871.916 ms 2024-04-27T02:21:22.551 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:22 smithi008 ceph-mon[63900]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:22.551 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:22 smithi008 ceph-mon[63900]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:22.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:22 smithi008 ceph-mon[63900]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T02:21:22.552 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:22 smithi008 ceph-mon[63900]: pgmap v227: 97 pgs: 2 remapped+peering, 17 peering, 78 active+clean; 358 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s wr, 210 op/s; 601 KiB/s, 63 keys/s, 34 objects/s recovering 2024-04-27T02:21:22.584 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:21:22.584 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:21:22.584 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-27T02:21:22.585 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 9 2024-04-27T02:21:22.586 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:21:22.586 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:21:22.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254385 9.78 MB/sec execute 509 sec latency 741.278 ms 2024-04-27T02:21:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:22 smithi077 ceph-mon[50325]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:22.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:22 smithi077 ceph-mon[50325]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:22.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:22 smithi077 ceph-mon[50325]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T02:21:22.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:22 smithi077 ceph-mon[50325]: pgmap v227: 97 pgs: 2 remapped+peering, 17 peering, 78 active+clean; 358 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s wr, 210 op/s; 601 KiB/s, 63 keys/s, 34 objects/s recovering 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:21:23.200 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:21:23.201 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "11/23 daemons upgraded", 2024-04-27T02:21:23.201 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:21:23.201 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:21:23.201 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:21:23.602 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:23 smithi008 ceph-mon[63900]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:23.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:23 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/1602 objects degraded (4.307%), 8 pgs degraded) 2024-04-27T02:21:23.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:23 smithi008 ceph-mon[63900]: osdmap e125: 8 total, 8 up, 8 in 2024-04-27T02:21:23.603 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:23 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3730555461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:23.808 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:21:23.808 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:21:23.808 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:21:23.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 254737 9.79 MB/sec execute 510 sec latency 47.714 ms 2024-04-27T02:21:23.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:23 smithi077 ceph-mon[50325]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:23.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:23 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/1602 objects degraded (4.307%), 8 pgs degraded) 2024-04-27T02:21:23.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:23 smithi077 ceph-mon[50325]: osdmap e125: 8 total, 8 up, 8 in 2024-04-27T02:21:23.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:23 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3730555461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:24.648 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:24 smithi008 ceph-mon[63900]: from='client.44277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:24.648 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:24 smithi008 ceph-mon[63900]: pgmap v229: 97 pgs: 2 peering, 95 active+clean; 357 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 560 op/s; 1.5 MiB/s, 61 keys/s, 68 objects/s recovering 2024-04-27T02:21:24.649 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:24 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/165733482' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:21:24.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:24 smithi077 ceph-mon[50325]: from='client.44277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:24.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:24 smithi077 ceph-mon[50325]: pgmap v229: 97 pgs: 2 peering, 95 active+clean; 357 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 560 op/s; 1.5 MiB/s, 61 keys/s, 68 objects/s recovering 2024-04-27T02:21:24.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:24 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/165733482' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:21:24.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255092 9.78 MB/sec execute 511 sec latency 75.705 ms 2024-04-27T02:21:25.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255439 9.78 MB/sec execute 512 sec latency 58.864 ms 2024-04-27T02:21:26.700 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:26 smithi008 ceph-mon[63900]: pgmap v230: 97 pgs: 2 peering, 95 active+clean; 356 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.8 MiB/s wr, 412 op/s; 894 KiB/s, 42 keys/s, 38 objects/s recovering 2024-04-27T02:21:26.904 INFO:teuthology.orchestra.run.smithi008.stdout: 5 255814 9.78 MB/sec execute 513 sec latency 41.962 ms 2024-04-27T02:21:26.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:26 smithi077 ceph-mon[50325]: pgmap v230: 97 pgs: 2 peering, 95 active+clean; 356 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.8 MiB/s wr, 412 op/s; 894 KiB/s, 42 keys/s, 38 objects/s recovering 2024-04-27T02:21:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256250 9.78 MB/sec execute 514 sec latency 37.831 ms 2024-04-27T02:21:28.729 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:28 smithi008 ceph-mon[63900]: pgmap v231: 97 pgs: 97 active+clean; 353 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.5 MiB/s wr, 464 op/s; 582 KiB/s, 20 objects/s recovering 2024-04-27T02:21:28.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 256649 9.79 MB/sec execute 515 sec latency 39.871 ms 2024-04-27T02:21:28.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:28 smithi077 ceph-mon[50325]: pgmap v231: 97 pgs: 97 active+clean; 353 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.5 MiB/s wr, 464 op/s; 582 KiB/s, 20 objects/s recovering 2024-04-27T02:21:29.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257075 9.80 MB/sec execute 516 sec latency 42.427 ms 2024-04-27T02:21:30.751 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:30 smithi008 ceph-mon[63900]: pgmap v232: 97 pgs: 97 active+clean; 358 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 563 op/s; 511 KiB/s, 17 objects/s recovering 2024-04-27T02:21:30.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257460 9.79 MB/sec execute 517 sec latency 35.462 ms 2024-04-27T02:21:30.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:30 smithi077 ceph-mon[50325]: pgmap v232: 97 pgs: 97 active+clean; 358 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.0 MiB/s wr, 563 op/s; 511 KiB/s, 17 objects/s recovering 2024-04-27T02:21:31.765 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:31.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:31 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:31.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:31 smithi008 ceph-mon[63900]: Upgrade: osd.5 is safe to restart 2024-04-27T02:21:31.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:31.766 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:31 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T02:21:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:31 smithi077 ceph-mon[50325]: Upgrade: osd.5 is safe to restart 2024-04-27T02:21:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:31.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 257760 9.78 MB/sec execute 518 sec latency 67.522 ms 2024-04-27T02:21:32.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:32 smithi008 ceph-mon[63900]: pgmap v233: 97 pgs: 97 active+clean; 361 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.8 MiB/s wr, 545 op/s; 416 KiB/s, 14 objects/s recovering 2024-04-27T02:21:32.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:32.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:21:32.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:32.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258030 9.78 MB/sec execute 519 sec latency 74.727 ms 2024-04-27T02:21:32.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:32 smithi077 ceph-mon[50325]: pgmap v233: 97 pgs: 97 active+clean; 361 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.8 MiB/s wr, 545 op/s; 416 KiB/s, 14 objects/s recovering 2024-04-27T02:21:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T02:21:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:33.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258380 9.79 MB/sec execute 520 sec latency 48.668 ms 2024-04-27T02:21:33.915 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:33 smithi008 ceph-mon[63900]: Upgrade: Updating osd.5 2024-04-27T02:21:33.915 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:33 smithi008 ceph-mon[63900]: Deploying daemon osd.5 on smithi077 2024-04-27T02:21:33.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:33 smithi077 ceph-mon[50325]: Upgrade: Updating osd.5 2024-04-27T02:21:33.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:33 smithi077 ceph-mon[50325]: Deploying daemon osd.5 on smithi077 2024-04-27T02:21:34.741 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:34 smithi077 ceph-mon[50325]: pgmap v234: 97 pgs: 97 active+clean; 355 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 529 op/s 2024-04-27T02:21:34.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 258759 9.78 MB/sec execute 521 sec latency 57.514 ms 2024-04-27T02:21:34.944 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:34 smithi008 ceph-mon[63900]: pgmap v234: 97 pgs: 97 active+clean; 355 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.0 MiB/s wr, 529 op/s 2024-04-27T02:21:35.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259009 9.77 MB/sec execute 522 sec latency 59.059 ms 2024-04-27T02:21:36.831 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:36 smithi077 systemd[1]: Stopping Ceph osd.5 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:21:36.831 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:36 smithi077 ceph-mon[50325]: pgmap v235: 97 pgs: 97 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-27T02:21:36.831 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:36 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:36.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259209 9.76 MB/sec execute 523 sec latency 65.044 ms 2024-04-27T02:21:36.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:36 smithi008 ceph-mon[63900]: pgmap v235: 97 pgs: 97 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-27T02:21:36.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:36 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:37.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:36 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[31418]: 2024-04-27T02:21:36.955+0000 7ff276c0a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:21:37.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:36 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[31418]: 2024-04-27T02:21:36.955+0000 7ff276c0a700 -1 osd.5 125 *** Got signal Terminated *** 2024-04-27T02:21:37.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:36 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[31418]: 2024-04-27T02:21:36.955+0000 7ff276c0a700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:21:37.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259452 9.76 MB/sec execute 524 sec latency 203.936 ms 2024-04-27T02:21:37.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:37 smithi008 ceph-mon[63900]: osd.5 marked itself down and dead 2024-04-27T02:21:37.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:37 smithi077 ceph-mon[50325]: osd.5 marked itself down and dead 2024-04-27T02:21:38.722 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:38 smithi077 podman[57092]: 2024-04-27 02:21:38.410563407 +0000 UTC m=+1.633457722 container died cdddbccab76c586fe7faeb13df91acbc9a447746581d9f3cdebc4650812d1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, release=754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-27T02:21:38.722 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:38 smithi077 podman[57092]: 2024-04-27 02:21:38.746072048 +0000 UTC m=+1.968966408 container cleanup cdddbccab76c586fe7faeb13df91acbc9a447746581d9f3cdebc4650812d1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , architecture=x86_64, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.expose-services=, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-27T02:21:38.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259478 9.74 MB/sec execute 525 sec latency 1063.851 ms 2024-04-27T02:21:38.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:38 smithi008 ceph-mon[63900]: pgmap v236: 97 pgs: 97 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 597 op/s 2024-04-27T02:21:38.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:38 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:21:38.980 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:38 smithi008 ceph-mon[63900]: osdmap e126: 8 total, 7 up, 8 in 2024-04-27T02:21:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:38 smithi077 ceph-mon[50325]: pgmap v236: 97 pgs: 97 active+clean; 354 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 597 op/s 2024-04-27T02:21:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:38 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:21:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:38 smithi077 ceph-mon[50325]: osdmap e126: 8 total, 7 up, 8 in 2024-04-27T02:21:38.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:38 smithi077 bash[57092]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5 2024-04-27T02:21:38.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:38 smithi077 podman[57106]: 2024-04-27 02:21:38.989292231 +0000 UTC m=+0.580198413 container remove cdddbccab76c586fe7faeb13df91acbc9a447746581d9f3cdebc4650812d1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, version=8) 2024-04-27T02:21:39.681 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:39 smithi077 podman[57198]: 2024-04-27 02:21:39.357338665 +0000 UTC m=+0.021744551 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:39.681 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:39 smithi077 podman[57198]: 2024-04-27 02:21:39.46580673 +0000 UTC m=+0.130212633 container create e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS) 2024-04-27T02:21:39.681 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:39 smithi077 podman[57198]: 2024-04-27 02:21:39.702117675 +0000 UTC m=+0.366523559 container init e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-27T02:21:39.682 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:39 smithi077 podman[57198]: 2024-04-27 02:21:39.705794404 +0000 UTC m=+0.370200274 container start e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-27T02:21:39.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 259861 9.75 MB/sec execute 526 sec latency 40.212 ms 2024-04-27T02:21:39.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:39 smithi077 ceph-mon[50325]: osdmap e127: 8 total, 7 up, 8 in 2024-04-27T02:21:39.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:39 smithi077 podman[57198]: 2024-04-27 02:21:39.793950347 +0000 UTC m=+0.458356224 container attach e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:21:39.995 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:39 smithi008 ceph-mon[63900]: osdmap e127: 8 total, 7 up, 8 in 2024-04-27T02:21:40.483 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:40 smithi077 podman[57198]: 2024-04-27 02:21:40.059775219 +0000 UTC m=+0.724181102 container died e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:21:40.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:40 smithi008 ceph-mon[63900]: pgmap v239: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 4 peering, 11 stale+active+clean, 79 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.2 MiB/s wr, 453 op/s; 24/1590 objects degraded (1.509%) 2024-04-27T02:21:40.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:40 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 24/1590 objects degraded (1.509%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:40.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260232 9.75 MB/sec execute 527 sec latency 54.328 ms 2024-04-27T02:21:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:40 smithi077 ceph-mon[50325]: pgmap v239: 97 pgs: 1 active+undersized, 2 active+undersized+degraded, 4 peering, 11 stale+active+clean, 79 active+clean; 355 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.2 MiB/s wr, 453 op/s; 24/1590 objects degraded (1.509%) 2024-04-27T02:21:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:40 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 24/1590 objects degraded (1.509%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:41.705 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 podman[57224]: 2024-04-27 02:21:41.407634147 +0000 UTC m=+1.342701130 container remove e41b345564a3d16948b080a175cee3877fc95d877fe6535cef78da0cc3fb0637 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:21:41.705 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-27T02:21:41.705 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 systemd[1]: Stopped Ceph osd.5 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:21:41.705 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.5.service: Consumed 5min 16.772s CPU time. 2024-04-27T02:21:41.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260605 9.76 MB/sec execute 528 sec latency 41.914 ms 2024-04-27T02:21:41.967 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 systemd[1]: Starting Ceph osd.5 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:21:41.968 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 podman[57323]: 2024-04-27 02:21:41.880285666 +0000 UTC m=+0.021857718 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:42.232 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:41 smithi077 podman[57323]: 2024-04-27 02:21:41.992210113 +0000 UTC m=+0.133782132 container create 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb) 2024-04-27T02:21:42.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 podman[57323]: 2024-04-27 02:21:42.228624345 +0000 UTC m=+0.370196377 container init 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:21:42.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 podman[57323]: 2024-04-27 02:21:42.232904631 +0000 UTC m=+0.374476652 container start 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS) 2024-04-27T02:21:42.700 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 podman[57323]: 2024-04-27 02:21:42.320310139 +0000 UTC m=+0.461882166 container attach 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-27T02:21:42.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:42 smithi008 ceph-mon[63900]: pgmap v240: 97 pgs: 4 active+undersized, 8 active+undersized+degraded, 4 peering, 7 stale+active+clean, 74 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.8 MiB/s wr, 409 op/s; 72/1617 objects degraded (4.453%) 2024-04-27T02:21:42.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 260907 9.74 MB/sec execute 529 sec latency 55.499 ms 2024-04-27T02:21:42.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-mon[50325]: pgmap v240: 97 pgs: 4 active+undersized, 8 active+undersized+degraded, 4 peering, 7 stale+active+clean, 74 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.8 MiB/s wr, 409 op/s; 72/1617 objects degraded (4.453%) 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 bash[57323]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-27T02:21:42.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T02:21:42.984 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate[57337]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-27T02:21:42.984 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:42 smithi077 podman[57323]: 2024-04-27 02:21:42.757315993 +0000 UTC m=+0.898888028 container died 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-27T02:21:43.869 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:43 smithi077 podman[57470]: 2024-04-27 02:21:43.54184224 +0000 UTC m=+0.779229203 container remove 7c105c6a07eca1aa132a542218c2178525473474de265ed850dd03eb250a53f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-27T02:21:43.869 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:43 smithi077 podman[57533]: 2024-04-27 02:21:43.665900367 +0000 UTC m=+0.021504798 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:21:43.870 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:43 smithi077 podman[57533]: 2024-04-27 02:21:43.776260263 +0000 UTC m=+0.131864675 container create a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:21:43.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261263 9.74 MB/sec execute 530 sec latency 46.833 ms 2024-04-27T02:21:44.232 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 podman[57533]: 2024-04-27 02:21:44.012256147 +0000 UTC m=+0.367860559 container init a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T02:21:44.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 podman[57533]: 2024-04-27 02:21:44.015778007 +0000 UTC m=+0.371382420 container start a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T02:21:44.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-osd[57551]: -- 172.21.15.77:0/4234545609 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8ee545680 con 0x55a8ee50f000 2024-04-27T02:21:44.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 bash[57533]: a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 2024-04-27T02:21:44.233 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 systemd[1]: Started Ceph osd.5 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:21:44.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:44 smithi008 ceph-mon[63900]: pgmap v241: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 469 op/s; 190/1635 objects degraded (11.621%) 2024-04-27T02:21:44.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:44.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:44.833 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:44 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:44.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261535 9.74 MB/sec execute 531 sec latency 62.613 ms 2024-04-27T02:21:44.983 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:21:44.561+0000 7f87f0abf7c0 -1 Falling back to public interface 2024-04-27T02:21:44.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-mon[50325]: pgmap v241: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 469 op/s; 190/1635 objects degraded (11.621%) 2024-04-27T02:21:44.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:44.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:44.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:44 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:45.852 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:45 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 191/1620 objects degraded (11.790%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:45.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 261897 9.74 MB/sec execute 532 sec latency 66.392 ms 2024-04-27T02:21:46.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:45 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 191/1620 objects degraded (11.790%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:46.863 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:46 smithi008 ceph-mon[63900]: pgmap v242: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.7 MiB/s wr, 304 op/s; 191/1620 objects degraded (11.790%) 2024-04-27T02:21:46.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:46.864 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:46.905 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262356 9.75 MB/sec execute 533 sec latency 41.833 ms 2024-04-27T02:21:47.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:46 smithi077 ceph-mon[50325]: pgmap v242: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.7 MiB/s wr, 304 op/s; 191/1620 objects degraded (11.790%) 2024-04-27T02:21:47.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:47.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:21:47.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 262695 9.74 MB/sec execute 534 sec latency 41.436 ms 2024-04-27T02:21:48.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:48 smithi008 ceph-mon[63900]: pgmap v243: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 463 op/s; 186/1626 objects degraded (11.439%) 2024-04-27T02:21:48.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:48 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:48.897 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:48 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:48.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263036 9.74 MB/sec execute 535 sec latency 43.040 ms 2024-04-27T02:21:49.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:48 smithi077 ceph-mon[50325]: pgmap v243: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 358 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 463 op/s; 186/1626 objects degraded (11.439%) 2024-04-27T02:21:49.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:48 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:49.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:48 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:49.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263405 9.75 MB/sec execute 536 sec latency 40.973 ms 2024-04-27T02:21:50.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263705 9.75 MB/sec execute 537 sec latency 59.250 ms 2024-04-27T02:21:50.920 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:50 smithi008 ceph-mon[63900]: pgmap v244: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 353 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 436 op/s; 176/1599 objects degraded (11.007%) 2024-04-27T02:21:50.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:50.921 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:51.022 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:50 smithi077 ceph-mon[50325]: pgmap v244: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 353 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 436 op/s; 176/1599 objects degraded (11.007%) 2024-04-27T02:21:51.022 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:51.022 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 263954 9.74 MB/sec execute 538 sec latency 53.884 ms 2024-04-27T02:21:51.986 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:51 smithi008 ceph-mon[63900]: pgmap v245: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 420 op/s; 179/1629 objects degraded (10.988%) 2024-04-27T02:21:51.986 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:51 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 179/1629 objects degraded (10.988%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:52.122 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:51 smithi077 ceph-mon[50325]: pgmap v245: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 351 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 420 op/s; 179/1629 objects degraded (10.988%) 2024-04-27T02:21:52.123 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:51 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 179/1629 objects degraded (10.988%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:52.733 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:52 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:21:52.580+0000 7f87f0abf7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-27T02:21:52.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264228 9.74 MB/sec execute 539 sec latency 51.209 ms 2024-04-27T02:21:53.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264484 9.72 MB/sec execute 540 sec latency 62.707 ms 2024-04-27T02:21:54.232 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:21:53.987+0000 7f87f0abf7c0 -1 osd.5 125 log_to_monitors true 2024-04-27T02:21:54.283 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:21:54.784 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: pgmap v246: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.1 MiB/s wr, 464 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.785 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:21:54.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264763 9.72 MB/sec execute 541 sec latency 65.882 ms 2024-04-27T02:21:54.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: pgmap v246: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.1 MiB/s wr, 464 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:54.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:21:55.267 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:21:55.267 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (8m) 82s ago 17m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:21:55.267 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (4m) 82s ago 17m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:21:55.267 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (4m) 6s ago 16m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:21:55.267 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 82s ago 17m - - 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (11m) 82s ago 11m 41.1M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (11m) 6s ago 11m 15.1M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (7m) 82s ago 18m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (6m) 6s ago 16m 451M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 82s ago 18m 53.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (5m) 6s ago 16m 42.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (11m) 82s ago 11m 286M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (11m) 6s ago 11m 88.8M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 82s ago 17m 9675k - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (7m) 6s ago 16m 17.4M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (3m) 82s ago 15m 366M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 82s ago 15m 292M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 82s ago 14m 222M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:21:55.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (91s) 82s ago 14m 15.1M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:21:55.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (49s) 6s ago 14m 276M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:21:55.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (11s) 6s ago 13m 12.6M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:21:55.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (13m) 6s ago 13m 546M 2704M 17.2.0 e1d6a67b021e a3456001b52f 2024-04-27T02:21:55.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (12m) 6s ago 12m 571M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:21:55.269 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (6m) 82s ago 16m 63.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:21:55.482 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:21:55.020+0000 7f87e1f17700 -1 osd.5 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:21:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 9 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:21:55.767 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:21:55.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: pgmap v247: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 421 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:55.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: pgmap v248: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 468 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:55.868 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: osdmap e128: 8 total, 7 up, 8 in 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:55.869 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:55 smithi008 ceph-mon[63900]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:55.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 264979 9.71 MB/sec execute 542 sec latency 90.845 ms 2024-04-27T02:21:55.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: pgmap v247: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 421 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:55.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: pgmap v248: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 468 op/s; 174/1638 objects degraded (10.623%) 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: osdmap e128: 8 total, 7 up, 8 in 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:55.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:55 smithi077 ceph-mon[50325]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:21:56.232 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "12/23 daemons upgraded", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:21:56.233 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:21:56.649 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 174/1638 objects degraded (10.623%), 23 pgs degraded 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 174/1638 objects degraded (10.623%), 23 pgs degraded 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-04-27T02:21:56.650 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-04-27T02:21:56.651 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-04-27T02:21:56.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265042 9.69 MB/sec execute 543 sec latency 901.806 ms 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2869983146' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949] boot 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:21:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:56 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2277060265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:21:56.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2869983146' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: osd.5 [v2:172.21.15.77:6808/2466004949,v1:172.21.15.77:6809/2466004949] boot 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T02:21:56.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:56 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2277060265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:21:57.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265081 9.68 MB/sec execute 544 sec latency 1901.852 ms 2024-04-27T02:21:57.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:57 smithi008 ceph-mon[63900]: pgmap v251: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 546 op/s; 175/1632 objects degraded (10.723%) 2024-04-27T02:21:57.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:57 smithi008 ceph-mon[63900]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:57.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:57 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 175/1632 objects degraded (10.723%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:57.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:57 smithi008 ceph-mon[63900]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T02:21:57.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:57 smithi077 ceph-mon[50325]: pgmap v251: 97 pgs: 10 active+undersized, 23 active+undersized+degraded, 64 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.6 MiB/s wr, 546 op/s; 175/1632 objects degraded (10.723%) 2024-04-27T02:21:57.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:57 smithi077 ceph-mon[50325]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:21:57.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:57 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 175/1632 objects degraded (10.723%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T02:21:57.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:57 smithi077 ceph-mon[50325]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T02:21:58.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265342 9.68 MB/sec execute 545 sec latency 2128.626 ms 2024-04-27T02:21:59.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:59 smithi008 ceph-mon[63900]: pgmap v253: 97 pgs: 5 remapped+peering, 7 active+undersized, 13 active+undersized+degraded, 72 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 593 op/s; 114/1545 objects degraded (7.379%); 26 KiB/s, 0 keys/s, 0 objects/s recovering 2024-04-27T02:21:59.160 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:21:59 smithi008 ceph-mon[63900]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T02:21:59.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:59 smithi077 ceph-mon[50325]: pgmap v253: 97 pgs: 5 remapped+peering, 7 active+undersized, 13 active+undersized+degraded, 72 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 593 op/s; 114/1545 objects degraded (7.379%); 26 KiB/s, 0 keys/s, 0 objects/s recovering 2024-04-27T02:21:59.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:21:59 smithi077 ceph-mon[50325]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T02:21:59.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265623 9.68 MB/sec execute 546 sec latency 69.558 ms 2024-04-27T02:22:00.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:00 smithi008 ceph-mon[63900]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T02:22:00.174 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:00 smithi008 ceph-mon[63900]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T02:22:00.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:00 smithi077 ceph-mon[50325]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T02:22:00.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:00 smithi077 ceph-mon[50325]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T02:22:00.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 265940 9.67 MB/sec execute 547 sec latency 325.688 ms 2024-04-27T02:22:01.753 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:01 smithi008 ceph-mon[63900]: pgmap v257: 97 pgs: 5 remapped+peering, 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 417 op/s; 110/1536 objects degraded (7.161%); 63 KiB/s, 0 keys/s, 11 objects/s recovering 2024-04-27T02:22:01.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:01 smithi008 ceph-mon[63900]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T02:22:01.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:01.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:01.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266170 9.66 MB/sec execute 548 sec latency 774.105 ms 2024-04-27T02:22:01.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:01 smithi077 ceph-mon[50325]: pgmap v257: 97 pgs: 5 remapped+peering, 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 77 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 417 op/s; 110/1536 objects degraded (7.161%); 63 KiB/s, 0 keys/s, 11 objects/s recovering 2024-04-27T02:22:01.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:01 smithi077 ceph-mon[50325]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T02:22:01.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:01.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:02.770 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:02 smithi008 ceph-mon[63900]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T02:22:02.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266424 9.66 MB/sec execute 549 sec latency 759.102 ms 2024-04-27T02:22:02.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:02 smithi077 ceph-mon[50325]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T02:22:03.543 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:03 smithi008 ceph-mon[63900]: pgmap v260: 97 pgs: 4 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s wr, 157 op/s; 52/1524 objects degraded (3.412%); 1.4 MiB/s, 28 objects/s recovering 2024-04-27T02:22:03.543 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:03 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 52/1524 objects degraded (3.412%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:03.544 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:03 smithi008 ceph-mon[63900]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T02:22:03.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266663 9.65 MB/sec execute 550 sec latency 44.297 ms 2024-04-27T02:22:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:03 smithi077 ceph-mon[50325]: pgmap v260: 97 pgs: 4 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.6 MiB/s wr, 157 op/s; 52/1524 objects degraded (3.412%); 1.4 MiB/s, 28 objects/s recovering 2024-04-27T02:22:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:03 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 52/1524 objects degraded (3.412%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:03 smithi077 ceph-mon[50325]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T02:22:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 266982 9.66 MB/sec execute 551 sec latency 51.089 ms 2024-04-27T02:22:05.650 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:05 smithi008 ceph-mon[63900]: pgmap v262: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.4 MiB/s wr, 228 op/s; 52/1554 objects degraded (3.346%); 2.4 MiB/s, 39 objects/s recovering 2024-04-27T02:22:05.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267205 9.65 MB/sec execute 552 sec latency 47.688 ms 2024-04-27T02:22:05.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:05 smithi077 ceph-mon[50325]: pgmap v262: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 345 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.4 MiB/s wr, 228 op/s; 52/1554 objects degraded (3.346%); 2.4 MiB/s, 39 objects/s recovering 2024-04-27T02:22:06.668 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:06 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 52/1554 objects degraded (3.346%), 2 pgs degraded) 2024-04-27T02:22:06.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267554 9.65 MB/sec execute 553 sec latency 52.613 ms 2024-04-27T02:22:06.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:06 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 52/1554 objects degraded (3.346%), 2 pgs degraded) 2024-04-27T02:22:07.693 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:07 smithi008 ceph-mon[63900]: pgmap v263: 97 pgs: 97 active+clean; 348 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.7 MiB/s wr, 281 op/s; 1.8 MiB/s, 31 objects/s recovering 2024-04-27T02:22:07.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 267866 9.65 MB/sec execute 554 sec latency 58.668 ms 2024-04-27T02:22:07.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:07 smithi077 ceph-mon[50325]: pgmap v263: 97 pgs: 97 active+clean; 348 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.7 MiB/s wr, 281 op/s; 1.8 MiB/s, 31 objects/s recovering 2024-04-27T02:22:08.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268159 9.64 MB/sec execute 555 sec latency 60.827 ms 2024-04-27T02:22:09.743 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:09 smithi008 ceph-mon[63900]: pgmap v264: 97 pgs: 97 active+clean; 349 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.9 MiB/s wr, 433 op/s; 686 KiB/s, 15 objects/s recovering 2024-04-27T02:22:09.743 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:22:09.880 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:09 smithi077 ceph-mon[50325]: pgmap v264: 97 pgs: 97 active+clean; 349 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.9 MiB/s wr, 433 op/s; 686 KiB/s, 15 objects/s recovering 2024-04-27T02:22:09.880 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:22:09.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268431 9.63 MB/sec execute 556 sec latency 54.166 ms 2024-04-27T02:22:10.761 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:10 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:22:10.762 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:10 smithi008 ceph-mon[63900]: Upgrade: osd.6 is safe to restart 2024-04-27T02:22:10.883 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:10 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T02:22:10.883 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:10 smithi077 ceph-mon[50325]: Upgrade: osd.6 is safe to restart 2024-04-27T02:22:10.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 268749 9.64 MB/sec execute 557 sec latency 40.450 ms 2024-04-27T02:22:11.799 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:11 smithi008 ceph-mon[63900]: pgmap v265: 97 pgs: 97 active+clean; 350 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 407 op/s; 606 KiB/s, 13 objects/s recovering 2024-04-27T02:22:11.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:11.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:22:11.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:11.800 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:11.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269047 9.63 MB/sec execute 558 sec latency 47.917 ms 2024-04-27T02:22:11.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:11 smithi077 ceph-mon[50325]: pgmap v265: 97 pgs: 97 active+clean; 350 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 407 op/s; 606 KiB/s, 13 objects/s recovering 2024-04-27T02:22:11.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:11.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T02:22:11.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:11.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:12.722 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:12 smithi008 ceph-mon[63900]: Upgrade: Updating osd.6 2024-04-27T02:22:12.723 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:12 smithi008 ceph-mon[63900]: Deploying daemon osd.6 on smithi077 2024-04-27T02:22:12.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:12 smithi077 ceph-mon[50325]: Upgrade: Updating osd.6 2024-04-27T02:22:12.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:12 smithi077 ceph-mon[50325]: Deploying daemon osd.6 on smithi077 2024-04-27T02:22:12.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269366 9.64 MB/sec execute 559 sec latency 43.206 ms 2024-04-27T02:22:13.755 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:13 smithi008 ceph-mon[63900]: pgmap v266: 97 pgs: 97 active+clean; 353 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 360 op/s; 512 KiB/s, 11 objects/s recovering 2024-04-27T02:22:13.779 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:13 smithi077 ceph-mon[50325]: pgmap v266: 97 pgs: 97 active+clean; 353 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 360 op/s; 512 KiB/s, 11 objects/s recovering 2024-04-27T02:22:13.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269609 9.62 MB/sec execute 560 sec latency 48.567 ms 2024-04-27T02:22:14.906 INFO:teuthology.orchestra.run.smithi008.stdout: 5 269903 9.62 MB/sec execute 561 sec latency 78.721 ms 2024-04-27T02:22:15.482 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:15 smithi077 systemd[1]: Stopping Ceph osd.6 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:22:15.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270068 9.60 MB/sec execute 562 sec latency 310.160 ms 2024-04-27T02:22:15.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:15 smithi008 ceph-mon[63900]: pgmap v267: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 428 op/s; 16 KiB/s, 4 objects/s recovering 2024-04-27T02:22:15.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:15 smithi008 ceph-mon[63900]: osd.6 marked itself down and dead 2024-04-27T02:22:15.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:15 smithi077 ceph-mon[50325]: pgmap v267: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 428 op/s; 16 KiB/s, 4 objects/s recovering 2024-04-27T02:22:15.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:15 smithi077 ceph-mon[50325]: osd.6 marked itself down and dead 2024-04-27T02:22:15.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[34397]: 2024-04-27T02:22:15.588+0000 7f064f7a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:22:15.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[34397]: 2024-04-27T02:22:15.588+0000 7f064f7a4700 -1 osd.6 136 *** Got signal Terminated *** 2024-04-27T02:22:15.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[34397]: 2024-04-27T02:22:15.588+0000 7f064f7a4700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:22:16.556 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:16 smithi077 podman[59353]: 2024-04-27 02:22:16.323201491 +0000 UTC m=+0.995049997 container died a3456001b52fd48a9f265904894d3cfe04bd8926fbc8a2f4606a04a9f112b017 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vcs-type=git, distribution-scope=public, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, RELEASE=HEAD) 2024-04-27T02:22:16.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270123 9.59 MB/sec execute 563 sec latency 1118.176 ms 2024-04-27T02:22:16.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:16 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:22:16.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:16 smithi008 ceph-mon[63900]: osdmap e137: 8 total, 7 up, 8 in 2024-04-27T02:22:16.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:16.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:16.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:16 smithi008 ceph-mon[63900]: osdmap e138: 8 total, 7 up, 8 in 2024-04-27T02:22:16.932 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:16 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:22:16.933 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:16 smithi077 ceph-mon[50325]: osdmap e137: 8 total, 7 up, 8 in 2024-04-27T02:22:16.933 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:16.933 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:16.933 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:16 smithi077 ceph-mon[50325]: osdmap e138: 8 total, 7 up, 8 in 2024-04-27T02:22:16.933 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:16 smithi077 podman[59353]: 2024-04-27 02:22:16.74678055 +0000 UTC m=+1.418629091 container cleanup a3456001b52fd48a9f265904894d3cfe04bd8926fbc8a2f4606a04a9f112b017 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, vendor=Red Hat, Inc., GIT_CLEAN=True, ceph=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, version=8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-04-27T02:22:16.933 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:16 smithi077 bash[59353]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6 2024-04-27T02:22:17.232 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:16 smithi077 podman[59372]: 2024-04-27 02:22:16.9566426 +0000 UTC m=+0.636383564 container remove a3456001b52fd48a9f265904894d3cfe04bd8926fbc8a2f4606a04a9f112b017 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., version=8, release=754, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-27T02:22:17.668 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:17 smithi077 podman[59458]: 2024-04-27 02:22:17.320546611 +0000 UTC m=+0.020161189 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:17.668 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:17 smithi077 podman[59458]: 2024-04-27 02:22:17.441450182 +0000 UTC m=+0.141064743 container create 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb) 2024-04-27T02:22:17.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:17 smithi008 ceph-mon[63900]: pgmap v269: 97 pgs: 10 stale+active+clean, 87 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 414 op/s 2024-04-27T02:22:17.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270409 9.59 MB/sec execute 564 sec latency 144.832 ms 2024-04-27T02:22:17.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:17 smithi077 ceph-mon[50325]: pgmap v269: 97 pgs: 10 stale+active+clean, 87 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 414 op/s 2024-04-27T02:22:17.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:17 smithi077 podman[59458]: 2024-04-27 02:22:17.727873538 +0000 UTC m=+0.427488099 container init 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:22:17.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:17 smithi077 podman[59458]: 2024-04-27 02:22:17.731255498 +0000 UTC m=+0.430870068 container start 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, ceph=True) 2024-04-27T02:22:17.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:17 smithi077 podman[59458]: 2024-04-27 02:22:17.827908528 +0000 UTC m=+0.527523099 container attach 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-27T02:22:18.420 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:18 smithi077 podman[59458]: 2024-04-27 02:22:18.073025412 +0000 UTC m=+0.772639966 container died 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-27T02:22:18.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 270832 9.60 MB/sec execute 565 sec latency 40.493 ms 2024-04-27T02:22:18.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:18 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 67/1569 objects degraded (4.270%), 10 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:18.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:18 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 67/1569 objects degraded (4.270%), 10 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:18.985 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:18 smithi077 podman[59484]: 2024-04-27 02:22:18.766710377 +0000 UTC m=+0.683118319 container remove 1b6e14f72f78eff69c9eb7a09339a0901ff898e463d4f029528c0d3ccb4f2ffa (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T02:22:18.985 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:18 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-27T02:22:18.986 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:18 smithi077 systemd[1]: Stopped Ceph osd.6 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:22:18.986 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:18 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.6.service: Consumed 4min 57.778s CPU time. 2024-04-27T02:22:19.268 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 systemd[1]: Starting Ceph osd.6 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:22:19.268 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 podman[59587]: 2024-04-27 02:22:19.160942617 +0000 UTC m=+0.020695888 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:19.554 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 podman[59587]: 2024-04-27 02:22:19.292890422 +0000 UTC m=+0.152643694 container create 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, GIT_CLEAN=True, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:22:19.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271250 9.61 MB/sec execute 566 sec latency 38.375 ms 2024-04-27T02:22:19.969 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:19 smithi008 ceph-mon[63900]: pgmap v271: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 8 stale+active+clean, 77 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 388 op/s; 67/1569 objects degraded (4.270%) 2024-04-27T02:22:19.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:19 smithi077 ceph-mon[50325]: pgmap v271: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 8 stale+active+clean, 77 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 388 op/s; 67/1569 objects degraded (4.270%) 2024-04-27T02:22:19.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 podman[59587]: 2024-04-27 02:22:19.57934375 +0000 UTC m=+0.439097033 container init 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T02:22:19.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 podman[59587]: 2024-04-27 02:22:19.584328524 +0000 UTC m=+0.444081793 container start 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T02:22:19.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:19 smithi077 podman[59587]: 2024-04-27 02:22:19.687777733 +0000 UTC m=+0.547531009 container attach 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 bash[59587]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T02:22:20.483 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T02:22:20.484 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-27T02:22:20.484 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T02:22:20.484 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate[59601]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-27T02:22:20.484 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:20 smithi077 podman[59587]: 2024-04-27 02:22:20.177209273 +0000 UTC m=+1.036962550 container died 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:22:20.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 271632 9.60 MB/sec execute 567 sec latency 32.104 ms 2024-04-27T02:22:21.545 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 podman[59734]: 2024-04-27 02:22:21.310278188 +0000 UTC m=+1.123719025 container remove 5df74adeef040d145ada5a1dbf09792a0a5d48387331350203eec43a85c6c173 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-activate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T02:22:21.545 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 podman[59793]: 2024-04-27 02:22:21.445161698 +0000 UTC m=+0.021657144 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:21.545 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 podman[59793]: 2024-04-27 02:22:21.569704262 +0000 UTC m=+0.146199698 container create d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:22:21.749 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:21 smithi008 ceph-mon[63900]: pgmap v272: 97 pgs: 3 active+undersized, 14 active+undersized+degraded, 6 stale+active+clean, 74 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.2 MiB/s wr, 402 op/s; 102/1605 objects degraded (6.355%) 2024-04-27T02:22:21.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:21 smithi077 ceph-mon[50325]: pgmap v272: 97 pgs: 3 active+undersized, 14 active+undersized+degraded, 6 stale+active+clean, 74 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.2 MiB/s wr, 402 op/s; 102/1605 objects degraded (6.355%) 2024-04-27T02:22:21.809 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 podman[59793]: 2024-04-27 02:22:21.797395654 +0000 UTC m=+0.373891091 container init d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-27T02:22:21.809 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 podman[59793]: 2024-04-27 02:22:21.800682978 +0000 UTC m=+0.377178417 container start d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T02:22:21.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272022 9.60 MB/sec execute 568 sec latency 57.189 ms 2024-04-27T02:22:22.086 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 ceph-osd[59811]: -- 172.21.15.77:0/2167883602 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624dd517680 con 0x5624dd4e0c00 2024-04-27T02:22:22.086 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 bash[59793]: d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 2024-04-27T02:22:22.086 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:21 smithi077 systemd[1]: Started Ceph osd.6 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:22:22.482 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:22:22.107+0000 7fca985d67c0 -1 Falling back to public interface 2024-04-27T02:22:22.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272366 9.60 MB/sec execute 569 sec latency 46.035 ms 2024-04-27T02:22:23.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:23.029 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:23.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:22:23.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:22 smithi008 ceph-mon[63900]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 401 op/s; 182/1620 objects degraded (11.235%) 2024-04-27T02:22:23.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:23.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:23.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:22:23.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:22 smithi077 ceph-mon[50325]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 401 op/s; 182/1620 objects degraded (11.235%) 2024-04-27T02:22:23.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 272754 9.61 MB/sec execute 570 sec latency 35.233 ms 2024-04-27T02:22:24.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:24 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 190/1656 objects degraded (11.473%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:24.296 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:24 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 190/1656 objects degraded (11.473%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:24.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273031 9.60 MB/sec execute 571 sec latency 66.538 ms 2024-04-27T02:22:25.316 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:25 smithi008 ceph-mon[63900]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.9 MiB/s wr, 395 op/s; 190/1656 objects degraded (11.473%) 2024-04-27T02:22:25.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:25 smithi077 ceph-mon[50325]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.9 MiB/s wr, 395 op/s; 190/1656 objects degraded (11.473%) 2024-04-27T02:22:25.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273472 9.60 MB/sec execute 572 sec latency 34.260 ms 2024-04-27T02:22:26.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 273750 9.60 MB/sec execute 573 sec latency 60.121 ms 2024-04-27T02:22:27.264 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:22:27.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:27 smithi008 ceph-mon[63900]: pgmap v275: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 458 op/s; 188/1599 objects degraded (11.757%) 2024-04-27T02:22:27.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:27.379 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:27.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:27 smithi077 ceph-mon[50325]: pgmap v275: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 458 op/s; 188/1599 objects degraded (11.757%) 2024-04-27T02:22:27.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:27.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274109 9.61 MB/sec execute 574 sec latency 49.066 ms 2024-04-27T02:22:28.350 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:22:28.350 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 115s ago 18m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:22:28.350 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 115s ago 17m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (5m) 2s ago 17m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 115s ago 17m - - 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (12m) 115s ago 12m 41.1M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (12m) 2s ago 12m 15.2M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (8m) 115s ago 19m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (7m) 2s ago 17m 451M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (6m) 115s ago 19m 53.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (6m) 2s ago 17m 43.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (11m) 115s ago 11m 286M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (11m) 2s ago 11m 88.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (8m) 115s ago 17m 9675k - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (8m) 2s ago 17m 20.5M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (4m) 115s ago 16m 366M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (3m) 115s ago 15m 292M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:22:28.351 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (2m) 115s ago 15m 222M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 115s ago 15m 15.1M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (82s) 2s ago 14m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (44s) 2s ago 14m 279M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (6s) 2s ago 13m 11.7M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (13m) 2s ago 13m 626M 2704M 17.2.0 e1d6a67b021e e255623daf74 2024-04-27T02:22:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 115s ago 16m 63.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:22:28.645 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:28 smithi008 ceph-mon[63900]: from='client.44303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:28.645 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:28 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:28.645 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:28 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:28.645 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:28 smithi008 ceph-mon[63900]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:28.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:28 smithi077 ceph-mon[50325]: from='client.44303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:28.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:28 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:28.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:28 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:28.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:28 smithi077 ceph-mon[50325]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:28.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274414 9.60 MB/sec execute 575 sec latency 50.322 ms 2024-04-27T02:22:29.008 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:22:29.008 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:22:29.008 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 6 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 10 2024-04-27T02:22:29.009 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:22:29.010 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:22:29.230 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:28 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:22:28.844+0000 7fca985d67c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:22:29.536 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "13/23 daemons upgraded", 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:22:29.537 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:22:29.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:29 smithi008 ceph-mon[63900]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.5 MiB/s wr, 582 op/s; 186/1605 objects degraded (11.589%) 2024-04-27T02:22:29.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:29 smithi008 ceph-mon[63900]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:29.680 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:29 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1310350827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:29.681 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:29 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 186/1605 objects degraded (11.589%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:29.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:29 smithi077 ceph-mon[50325]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 347 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.5 MiB/s wr, 582 op/s; 186/1605 objects degraded (11.589%) 2024-04-27T02:22:29.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:29 smithi077 ceph-mon[50325]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:29.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:29 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1310350827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:29.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:29 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 186/1605 objects degraded (11.589%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:29.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 274747 9.60 MB/sec execute 576 sec latency 60.208 ms 2024-04-27T02:22:30.188 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 186/1605 objects degraded (11.589%), 25 pgs degraded 2024-04-27T02:22:30.188 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: osd.6 (root=default,host=smithi077) is down 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 186/1605 objects degraded (11.589%), 25 pgs degraded 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-04-27T02:22:30.189 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-04-27T02:22:30.190 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-04-27T02:22:30.191 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-04-27T02:22:30.482 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:22:30.050+0000 7fca985d67c0 -1 osd.6 136 log_to_monitors true 2024-04-27T02:22:30.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:30 smithi008 ceph-mon[63900]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:30.686 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:30 smithi008 ceph-mon[63900]: from='osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:22:30.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:30 smithi008 ceph-mon[63900]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:22:30.687 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:30 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3251401832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:22:30.902 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:22:30.561+0000 7fca89a2e700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:22:30.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-mon[50325]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:22:30.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-mon[50325]: from='osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:22:30.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-mon[50325]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T02:22:30.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:30 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3251401832' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:22:30.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275010 9.59 MB/sec execute 577 sec latency 49.876 ms 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: pgmap v277: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 501 op/s; 185/1590 objects degraded (11.635%) 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: osdmap e139: 8 total, 7 up, 8 in 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: from='osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:31.760 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:31.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275170 9.58 MB/sec execute 578 sec latency 413.041 ms 2024-04-27T02:22:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: pgmap v277: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.0 MiB/s wr, 501 op/s; 185/1590 objects degraded (11.635%) 2024-04-27T02:22:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T02:22:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: osdmap e139: 8 total, 7 up, 8 in 2024-04-27T02:22:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: from='osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:22:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:22:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:32.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:22:32.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233] boot 2024-04-27T02:22:32.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T02:22:32.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: pgmap v280: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.0 MiB/s wr, 497 op/s; 184/1596 objects degraded (11.529%) 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: pgmap v281: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 445 op/s; 184/1596 objects degraded (11.529%) 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:22:32.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T02:22:32.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:32 smithi008 ceph-mon[63900]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T02:22:32.909 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275170 9.57 MB/sec execute 579 sec latency 1413.069 ms 2024-04-27T02:22:32.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: osd.6 [v2:172.21.15.77:6816/2655276233,v1:172.21.15.77:6817/2655276233] boot 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: pgmap v280: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.0 MiB/s wr, 497 op/s; 184/1596 objects degraded (11.529%) 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: pgmap v281: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.2 MiB/s wr, 445 op/s; 184/1596 objects degraded (11.529%) 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:22:32.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:32.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:32.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T02:22:32.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:32 smithi077 ceph-mon[50325]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T02:22:33.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275170 9.55 MB/sec execute 580 sec latency 2413.129 ms 2024-04-27T02:22:34.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:34 smithi008 ceph-mon[63900]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T02:22:34.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:34 smithi008 ceph-mon[63900]: pgmap v284: 97 pgs: 1 remapped+peering, 10 peering, 8 active+undersized, 16 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s wr, 472 op/s; 119/1626 objects degraded (7.319%) 2024-04-27T02:22:34.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:34 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 184/1596 objects degraded (11.529%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:34.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:34 smithi008 ceph-mon[63900]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T02:22:34.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275293 9.54 MB/sec execute 581 sec latency 2570.400 ms 2024-04-27T02:22:34.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:34 smithi077 ceph-mon[50325]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T02:22:34.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:34 smithi077 ceph-mon[50325]: pgmap v284: 97 pgs: 1 remapped+peering, 10 peering, 8 active+undersized, 16 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s wr, 472 op/s; 119/1626 objects degraded (7.319%) 2024-04-27T02:22:34.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:34 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 184/1596 objects degraded (11.529%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:34.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:34 smithi077 ceph-mon[50325]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T02:22:35.910 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275353 9.53 MB/sec execute 582 sec latency 1064.498 ms 2024-04-27T02:22:36.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:36 smithi008 ceph-mon[63900]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T02:22:36.587 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:36 smithi008 ceph-mon[63900]: pgmap v287: 97 pgs: 13 peering, 84 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.1 MiB/s wr, 425 op/s; 1.5 MiB/s, 69 keys/s, 33 objects/s recovering 2024-04-27T02:22:36.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275689 9.52 MB/sec execute 583 sec latency 41.035 ms 2024-04-27T02:22:36.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:36 smithi077 ceph-mon[50325]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T02:22:36.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:36 smithi077 ceph-mon[50325]: pgmap v287: 97 pgs: 13 peering, 84 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.1 MiB/s wr, 425 op/s; 1.5 MiB/s, 69 keys/s, 33 objects/s recovering 2024-04-27T02:22:37.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:37 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 119/1626 objects degraded (7.319%), 16 pgs degraded) 2024-04-27T02:22:37.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 275980 9.52 MB/sec execute 584 sec latency 56.488 ms 2024-04-27T02:22:37.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:37 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 119/1626 objects degraded (7.319%), 16 pgs degraded) 2024-04-27T02:22:38.618 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:38 smithi008 ceph-mon[63900]: pgmap v288: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 191 op/s; 3.6 MiB/s, 54 keys/s, 42 objects/s recovering 2024-04-27T02:22:38.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276279 9.52 MB/sec execute 585 sec latency 37.958 ms 2024-04-27T02:22:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:38 smithi077 ceph-mon[50325]: pgmap v288: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 191 op/s; 3.6 MiB/s, 54 keys/s, 42 objects/s recovering 2024-04-27T02:22:39.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276567 9.52 MB/sec execute 586 sec latency 54.555 ms 2024-04-27T02:22:40.898 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:40 smithi008 ceph-mon[63900]: pgmap v289: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.1 MiB/s wr, 178 op/s; 3.0 MiB/s, 45 keys/s, 35 objects/s recovering 2024-04-27T02:22:40.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 276988 9.52 MB/sec execute 587 sec latency 57.734 ms 2024-04-27T02:22:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:40 smithi077 ceph-mon[50325]: pgmap v289: 97 pgs: 3 peering, 94 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.1 MiB/s wr, 178 op/s; 3.0 MiB/s, 45 keys/s, 35 objects/s recovering 2024-04-27T02:22:41.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277231 9.51 MB/sec execute 588 sec latency 80.210 ms 2024-04-27T02:22:42.718 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:42 smithi008 ceph-mon[63900]: pgmap v290: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 173 op/s; 2.3 MiB/s, 34 keys/s, 27 objects/s recovering 2024-04-27T02:22:42.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277566 9.52 MB/sec execute 589 sec latency 83.877 ms 2024-04-27T02:22:42.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:42 smithi077 ceph-mon[50325]: pgmap v290: 97 pgs: 97 active+clean; 344 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 173 op/s; 2.3 MiB/s, 34 keys/s, 27 objects/s recovering 2024-04-27T02:22:43.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 277916 9.52 MB/sec execute 590 sec latency 44.097 ms 2024-04-27T02:22:44.754 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:44 smithi008 ceph-mon[63900]: pgmap v291: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.4 MiB/s wr, 348 op/s; 2.0 MiB/s, 30 keys/s, 23 objects/s recovering 2024-04-27T02:22:44.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278308 9.52 MB/sec execute 591 sec latency 51.982 ms 2024-04-27T02:22:44.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:44 smithi077 ceph-mon[50325]: pgmap v291: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.4 MiB/s wr, 348 op/s; 2.0 MiB/s, 30 keys/s, 23 objects/s recovering 2024-04-27T02:22:45.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278601 9.51 MB/sec execute 592 sec latency 42.234 ms 2024-04-27T02:22:46.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:46 smithi008 ceph-mon[63900]: pgmap v292: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.9 MiB/s wr, 397 op/s; 1.6 MiB/s, 11 objects/s recovering 2024-04-27T02:22:46.786 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:46.787 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:46.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 278971 9.51 MB/sec execute 593 sec latency 56.774 ms 2024-04-27T02:22:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:46 smithi077 ceph-mon[50325]: pgmap v292: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.9 MiB/s wr, 397 op/s; 1.6 MiB/s, 11 objects/s recovering 2024-04-27T02:22:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:22:47.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:47 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:47.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:47 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:47.793 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:47 smithi008 ceph-mon[63900]: Upgrade: osd.7 is safe to restart 2024-04-27T02:22:47.840 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:47 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:47.840 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:47 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T02:22:47.841 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:47 smithi077 ceph-mon[50325]: Upgrade: osd.7 is safe to restart 2024-04-27T02:22:47.907 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279426 9.52 MB/sec execute 594 sec latency 52.300 ms 2024-04-27T02:22:48.805 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:48 smithi008 ceph-mon[63900]: pgmap v293: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 MiB/s wr, 384 op/s; 1.0 MiB/s, 6 objects/s recovering 2024-04-27T02:22:48.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 279894 9.53 MB/sec execute 595 sec latency 77.485 ms 2024-04-27T02:22:48.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:48 smithi077 ceph-mon[50325]: pgmap v293: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 MiB/s wr, 384 op/s; 1.0 MiB/s, 6 objects/s recovering 2024-04-27T02:22:49.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280121 9.52 MB/sec execute 596 sec latency 92.355 ms 2024-04-27T02:22:50.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: Upgrade: Updating osd.7 2024-04-27T02:22:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:22:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: Deploying daemon osd.7 on smithi077 2024-04-27T02:22:50.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:50 smithi077 ceph-mon[50325]: pgmap v294: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.2 MiB/s wr, 387 op/s 2024-04-27T02:22:50.337 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: Upgrade: Updating osd.7 2024-04-27T02:22:50.337 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:50.337 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T02:22:50.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:22:50.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: Deploying daemon osd.7 on smithi077 2024-04-27T02:22:50.338 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:50 smithi008 ceph-mon[63900]: pgmap v294: 97 pgs: 97 active+clean; 342 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.2 MiB/s wr, 387 op/s 2024-04-27T02:22:50.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280499 9.51 MB/sec execute 597 sec latency 51.224 ms 2024-04-27T02:22:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280716 9.51 MB/sec execute 598 sec latency 70.985 ms 2024-04-27T02:22:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:52 smithi077 ceph-mon[50325]: pgmap v295: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 529 op/s 2024-04-27T02:22:52.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:22:52.838 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:52 smithi008 ceph-mon[63900]: pgmap v295: 97 pgs: 97 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 529 op/s 2024-04-27T02:22:52.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 280944 9.50 MB/sec execute 599 sec latency 47.881 ms 2024-04-27T02:22:52.982 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:52 smithi077 systemd[1]: Stopping Ceph osd.7 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:22:53.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[37493]: 2024-04-27T02:22:53.051+0000 7fafb0c59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:22:53.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[37493]: 2024-04-27T02:22:53.051+0000 7fafb0c59700 -1 osd.7 144 *** Got signal Terminated *** 2024-04-27T02:22:53.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[37493]: 2024-04-27T02:22:53.051+0000 7fafb0c59700 -1 osd.7 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:22:53.594 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:53 smithi008 ceph-mon[63900]: osd.7 marked itself down and dead 2024-04-27T02:22:53.908 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 600 sec 2024-04-27T02:22:53.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:53 smithi077 ceph-mon[50325]: osd.7 marked itself down and dead 2024-04-27T02:22:54.520 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:54 smithi077 podman[61592]: 2024-04-27 02:22:54.281615284 +0000 UTC m=+1.353612277 container died e255623daf742437128c6ea4be2a0afc9aa0f3c93e6d0d80eb4905f834f99e5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, release=754, name=centos-stream, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vendor=Red Hat, Inc., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git) 2024-04-27T02:22:54.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:54 smithi008 ceph-mon[63900]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:22:54.815 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:54 smithi008 ceph-mon[63900]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-27T02:22:54.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:54 smithi008 ceph-mon[63900]: osdmap e145: 8 total, 7 up, 8 in 2024-04-27T02:22:54.816 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:54 smithi008 ceph-mon[63900]: pgmap v297: 97 pgs: 13 stale+active+clean, 84 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 461 op/s 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 601 sec 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 182792 4.753 2570.395 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Close 134234 0.148 15.745 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 7756 10.321 1118.165 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 36971 7.261 2518.171 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 165790 1.213 197.559 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 28917 0.007 1.492 2024-04-27T02:22:54.853 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 30438 1.449 4657.682 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 14906 12.967 1053.825 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: Find 64085 1.068 22.437 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 90405 13.842 2529.728 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 287180 0.011 3.515 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: LockX 600 0.427 2.746 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: UnlockX 600 0.354 2.391 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 12816 0.003 0.035 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:22:54.854 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 9.49977 MB/sec 5 clients 5 procs max_latency=4657.695 ms 2024-04-27T02:22:54.856 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-04-27T02:22:54.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-04-27T02:22:54.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:54 smithi077 ceph-mon[50325]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T02:22:54.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:54 smithi077 ceph-mon[50325]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-27T02:22:54.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:54 smithi077 ceph-mon[50325]: osdmap e145: 8 total, 7 up, 8 in 2024-04-27T02:22:54.912 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:54 smithi077 ceph-mon[50325]: pgmap v297: 97 pgs: 13 stale+active+clean, 84 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 461 op/s 2024-04-27T02:22:54.912 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:54 smithi077 podman[61592]: 2024-04-27 02:22:54.635571399 +0000 UTC m=+1.707568396 container cleanup e255623daf742437128c6ea4be2a0afc9aa0f3c93e6d0d80eb4905f834f99e5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-27T02:22:54.912 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:54 smithi077 bash[61592]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7 2024-04-27T02:22:55.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-04-27T02:22:55.081 INFO:teuthology.orchestra.run.smithi008.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-27T02:22:55.081 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:22:55.081 INFO:teuthology.orchestra.run.smithi008.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-04-27T02:22:55.082 INFO:teuthology.orchestra.run.smithi008.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-27T02:22:55.182 INFO:teuthology.orchestra.run.smithi008.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-27T02:22:55.182 INFO:teuthology.orchestra.run.smithi008.stdout:releasing clients 2024-04-27T02:22:55.233 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:54 smithi077 podman[61611]: 2024-04-27 02:22:54.936449367 +0000 UTC m=+0.658635041 container remove e255623daf742437128c6ea4be2a0afc9aa0f3c93e6d0d80eb4905f834f99e5a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, architecture=x86_64, GIT_CLEAN=True, distribution-scope=public, RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream) 2024-04-27T02:22:55.566 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.200385126 +0000 UTC m=+0.018113737 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:55.566 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.329280798 +0000 UTC m=+0.147009391 container create f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T02:22:55.707 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:55 smithi008 ceph-mon[63900]: osdmap e146: 8 total, 7 up, 8 in 2024-04-27T02:22:55.911 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.591041155 +0000 UTC m=+0.408769747 container init f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T02:22:55.911 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.59454566 +0000 UTC m=+0.412274262 container start f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-27T02:22:55.911 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.682347821 +0000 UTC m=+0.500076421 container attach f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-27T02:22:55.911 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:55 smithi077 ceph-mon[50325]: osdmap e146: 8 total, 7 up, 8 in 2024-04-27T02:22:56.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:55 smithi077 podman[61734]: 2024-04-27 02:22:55.936065407 +0000 UTC m=+0.753794007 container died f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T02:22:56.742 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:56 smithi008 ceph-mon[63900]: pgmap v299: 97 pgs: 6 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 62 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 519 op/s; 178/1512 objects degraded (11.772%) 2024-04-27T02:22:56.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:56 smithi077 ceph-mon[50325]: pgmap v299: 97 pgs: 6 active+undersized, 28 active+undersized+degraded, 1 stale+active+clean, 62 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.1 MiB/s wr, 519 op/s; 178/1512 objects degraded (11.772%) 2024-04-27T02:22:56.983 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:56 smithi077 podman[61760]: 2024-04-27 02:22:56.612111493 +0000 UTC m=+0.668618889 container remove f984b05ecc2f32c2a8e037129b643d4af138f50037ac77f5ee34781da219f699 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:22:56.983 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:56 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-27T02:22:56.983 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:56 smithi077 systemd[1]: Stopped Ceph osd.7 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:22:56.983 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:56 smithi077 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.7.service: Consumed 7min 28.206s CPU time. 2024-04-27T02:22:56.983 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:56 smithi077 systemd[1]: Starting Ceph osd.7 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:22:57.182 INFO:teuthology.orchestra.run.smithi008.stdout: 5 321 20.55 MB/sec execute 1 sec latency 77.140 ms 2024-04-27T02:22:57.445 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:57 smithi077 podman[61849]: 2024-04-27 02:22:57.053944984 +0000 UTC m=+0.024015155 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:57.446 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:57 smithi077 podman[61849]: 2024-04-27 02:22:57.188401405 +0000 UTC m=+0.158471548 container create 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-27T02:22:57.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:57 smithi077 ceph-mon[50325]: Health check failed: Degraded data redundancy: 178/1512 objects degraded (11.772%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:57.732 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:57 smithi077 podman[61849]: 2024-04-27 02:22:57.608385372 +0000 UTC m=+0.578455509 container init 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-27T02:22:57.733 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:57 smithi077 podman[61849]: 2024-04-27 02:22:57.616215272 +0000 UTC m=+0.586285415 container start 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T02:22:57.733 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:57 smithi077 podman[61849]: 2024-04-27 02:22:57.708427336 +0000 UTC m=+0.678497487 container attach 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T02:22:57.781 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:57 smithi008 ceph-mon[63900]: Health check failed: Degraded data redundancy: 178/1512 objects degraded (11.772%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T02:22:58.182 INFO:teuthology.orchestra.run.smithi008.stdout: 5 489 23.93 MB/sec execute 2 sec latency 27.201 ms 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-27T02:22:58.483 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 bash[61849]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate[61864]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-27T02:22:58.484 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 podman[61849]: 2024-04-27 02:22:58.21807938 +0000 UTC m=+1.188149522 container died 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-27T02:22:58.810 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:22:58 smithi008 ceph-mon[63900]: pgmap v300: 97 pgs: 7 active+undersized, 34 active+undersized+degraded, 56 active+clean; 321 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.9 MiB/s wr, 444 op/s; 198/1371 objects degraded (14.442%) 2024-04-27T02:22:58.872 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:22:58 smithi077 ceph-mon[50325]: pgmap v300: 97 pgs: 7 active+undersized, 34 active+undersized+degraded, 56 active+clean; 321 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.9 MiB/s wr, 444 op/s; 198/1371 objects degraded (14.442%) 2024-04-27T02:22:59.140 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:58 smithi077 podman[61996]: 2024-04-27 02:22:58.896667951 +0000 UTC m=+0.672621390 container remove 2da8d2a6d52e9a232daf6ef67c0a5eb9ec3368ab62429b591b5ced56ca337d72 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-27T02:22:59.140 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 podman[62051]: 2024-04-27 02:22:59.037228348 +0000 UTC m=+0.023486165 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:22:59.140 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 podman[62051]: 2024-04-27 02:22:59.164716227 +0000 UTC m=+0.150974031 container create 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0) 2024-04-27T02:22:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: 5 789 23.18 MB/sec execute 3 sec latency 38.367 ms 2024-04-27T02:22:59.413 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 podman[62051]: 2024-04-27 02:22:59.409201059 +0000 UTC m=+0.395458874 container init 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-27T02:22:59.413 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 podman[62051]: 2024-04-27 02:22:59.412618768 +0000 UTC m=+0.398876574 container start 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-27T02:22:59.732 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 ceph-osd[62070]: -- 172.21.15.77:0/507070388 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562018e39680 con 0x562018e02c00 2024-04-27T02:22:59.732 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 bash[62051]: 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 2024-04-27T02:22:59.732 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 systemd[1]: Started Ceph osd.7 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:23:00.183 INFO:teuthology.orchestra.run.smithi008.stdout: 5 1143 19.33 MB/sec execute 4 sec latency 35.744 ms 2024-04-27T02:23:00.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:22:59 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:22:59.955+0000 7f9fffa257c0 -1 Falling back to public interface 2024-04-27T02:23:00.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:00 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:00.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:00 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:00.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:00 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:00.595 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:00 smithi008 ceph-mon[63900]: pgmap v301: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 324 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.4 MiB/s wr, 350 op/s; 191/1311 objects degraded (14.569%) 2024-04-27T02:23:00.811 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:23:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:00 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:00 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:00.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:00 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:00.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:00 smithi077 ceph-mon[50325]: pgmap v301: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 324 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.4 MiB/s wr, 350 op/s; 191/1311 objects degraded (14.569%) 2024-04-27T02:23:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: 5 cleanup 5 sec 2024-04-27T02:23:01.351 INFO:teuthology.orchestra.run.smithi008.stdout:[1546] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c33c00000033' failed - Device or resource busy 2024-04-27T02:23:01.355 INFO:teuthology.orchestra.run.smithi008.stdout:[1546] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c34c00000035' failed - Device or resource busy 2024-04-27T02:23:01.355 INFO:teuthology.orchestra.run.smithi008.stdout:[1546] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c34700000022' failed - Device or resource busy 2024-04-27T02:23:01.360 INFO:teuthology.orchestra.run.smithi008.stdout:[1546] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c43a0000002a' failed - Device or resource busy 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: 0 cleanup 5 sec 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: Operation Count AvgLat MaxLat 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: ---------------------------------------- 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: NTCreateX 1058 6.510 59.349 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: Close 914 6.477 59.850 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: Rename 50 12.181 38.360 2024-04-27T02:23:01.408 INFO:teuthology.orchestra.run.smithi008.stdout: Unlink 104 3.874 21.364 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Qpathinfo 664 1.713 25.005 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Qfileinfo 409 14.821 57.375 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Qfsinfo 165 0.503 4.342 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Sfileinfo 170 12.681 26.705 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Find 279 0.950 10.598 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: WriteX 1809 0.024 0.096 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: ReadX 1226 0.012 0.292 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: Flush 125 12.068 21.891 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:23:01.409 INFO:teuthology.orchestra.run.smithi008.stdout:Throughput 19.3345 MB/sec 5 clients 5 procs max_latency=77.140 ms 2024-04-27T02:23:01.410 DEBUG:teuthology.parallel:result is None 2024-04-27T02:23:01.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:01 smithi008 ceph-mon[63900]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:01.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:01.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (9m) 2m ago 18m 16.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (5m) 2m ago 18m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (5m) 35s ago 17m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 2m ago 18m - - 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (12m) 2m ago 12m 41.1M - 17.2.0 e1d6a67b021e 05a5dc9fec6a 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (12m) 35s ago 12m 15.2M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:23:01.699 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (7m) 35s ago 17m 451M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 2m ago 19m 53.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (6m) 35s ago 17m 43.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (12m) 2m ago 12m 286M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (12m) 35s ago 12m 88.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 2m ago 18m 9675k - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (8m) 35s ago 17m 20.5M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 2m ago 17m 366M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 2m ago 16m 292M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (3m) 2m ago 16m 222M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (2m) 2m ago 15m 15.1M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (116s) 35s ago 15m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (78s) 35s ago 14m 279M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (40s) 35s ago 14m 11.7M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:23:01.700 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 starting - - - 2704M 2024-04-27T02:23:01.701 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (7m) 2m ago 17m 63.3M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:23:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:01 smithi077 ceph-mon[50325]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:02.129 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 7 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T02:23:02.130 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:02.131 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:23:02.131 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-27T02:23:02.131 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 11 2024-04-27T02:23:02.131 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:23:02.131 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "13/23 daemons upgraded", 2024-04-27T02:23:02.478 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T02:23:02.479 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:23:02.479 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:23:02.605 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:02 smithi077 ceph-mon[50325]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:02.605 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:02 smithi077 ceph-mon[50325]: pgmap v302: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 316 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 550 op/s; 184/1272 objects degraded (14.465%) 2024-04-27T02:23:02.605 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:02 smithi077 ceph-mon[50325]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:02.605 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:02 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1086019355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:02 smithi008 ceph-mon[63900]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:02 smithi008 ceph-mon[63900]: pgmap v302: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 316 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 15 MiB/s wr, 550 op/s; 184/1272 objects degraded (14.465%) 2024-04-27T02:23:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:02 smithi008 ceph-mon[63900]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:02 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1086019355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 184/1272 objects degraded (14.465%), 35 pgs degraded 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: osd.7 (root=default,host=smithi077) is down 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 184/1272 objects degraded (14.465%), 35 pgs degraded 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-04-27T02:23:02.963 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-04-27T02:23:02.964 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-27T02:23:02.965 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-04-27T02:23:02.966 INFO:teuthology.orchestra.run.smithi008.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-27T02:23:03.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:03 smithi008 ceph-mon[63900]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:03.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:03 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 184/1272 objects degraded (14.465%), 35 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:03.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:03 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1189189501' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:23:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:03 smithi077 ceph-mon[50325]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:03 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 184/1272 objects degraded (14.465%), 35 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:03.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:03 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1189189501' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:23:04.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:04 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:04.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:04 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:04.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:04 smithi008 ceph-mon[63900]: pgmap v303: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 13 MiB/s wr, 460 op/s; 202/1401 objects degraded (14.418%) 2024-04-27T02:23:04.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:04 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:04.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:04 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:04.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:04 smithi077 ceph-mon[50325]: pgmap v303: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 13 MiB/s wr, 460 op/s; 202/1401 objects degraded (14.418%) 2024-04-27T02:23:05.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:05.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:06.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:06.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:06.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:06 smithi077 ceph-mon[50325]: pgmap v304: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 259 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 12 MiB/s wr, 479 op/s; 112/723 objects degraded (15.491%) 2024-04-27T02:23:07.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:06 smithi008 ceph-mon[63900]: pgmap v304: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 259 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 12 MiB/s wr, 479 op/s; 112/723 objects degraded (15.491%) 2024-04-27T02:23:07.795 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:23:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:23:07.735+0000 7f9fffa257c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-27T02:23:07.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:07 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 82/513 objects degraded (15.984%), 32 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:08.208 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:07 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 82/513 objects degraded (15.984%), 32 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: pgmap v305: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 10 MiB/s wr, 348 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:09.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-27T02:23:09.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-27T02:23:09.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-27T02:23:09.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:08 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: pgmap v305: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 10 MiB/s wr, 348 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-27T02:23:09.180 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:08 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-27T02:23:09.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:23:09.202+0000 7f9fffa257c0 -1 osd.7 144 log_to_monitors true 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: pgmap v306: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 9.9 MiB/s wr, 366 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: pgmap v307: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 9.3 MiB/s wr, 328 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all osd 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-27T02:23:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: Upgrade: Setting require_osd_release to 19 squid 2024-04-27T02:23:10.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-27T02:23:10.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:23:10.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:09 smithi008 ceph-mon[63900]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:23:10.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: pgmap v306: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 9.9 MiB/s wr, 366 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: pgmap v307: 97 pgs: 10 active+undersized, 32 active+undersized+degraded, 55 active+clean; 236 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 9.3 MiB/s wr, 328 op/s; 82/513 objects degraded (15.984%) 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all osd 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: Upgrade: Setting require_osd_release to 19 squid 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:23:10.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:09 smithi077 ceph-mon[50325]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T02:23:10.732 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:23:10.326+0000 7f9fece75700 -1 osd.7 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T02:23:11.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-27T02:23:11.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-27T02:23:11.037 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: osdmap e147: 8 total, 7 up, 8 in 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.wzyiku"]}]: dispatch 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: Upgrade: It appears safe to stop mds.foofs.smithi008.wzyiku 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:23:11.038 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:10 smithi008 ceph-mon[63900]: from='osd.7 ' entity='osd.7' 2024-04-27T02:23:11.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: osdmap e147: 8 total, 7 up, 8 in 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi008.wzyiku"]}]: dispatch 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: Upgrade: It appears safe to stop mds.foofs.smithi008.wzyiku 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-04-27T02:23:11.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:10 smithi077 ceph-mon[50325]: from='osd.7 ' entity='osd.7' 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: pgmap v309: 97 pgs: 16 active+undersized, 26 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 683 KiB/s wr, 117 op/s; 73/426 objects degraded (17.136%) 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301] boot 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: osdmap e148: 8 total, 8 up, 8 in 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:23:12.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:11 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:12.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: pgmap v309: 97 pgs: 16 active+undersized, 26 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 683 KiB/s wr, 117 op/s; 73/426 objects degraded (17.136%) 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: osd.7 [v2:172.21.15.77:6824/2546079301,v1:172.21.15.77:6825/2546079301] boot 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: osdmap e148: 8 total, 8 up, 8 in 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi008.wzyiku", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:23:12.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:11 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:13.004 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:12 smithi008 ceph-mon[63900]: Upgrade: Updating mds.foofs.smithi008.wzyiku 2024-04-27T02:23:13.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:12 smithi008 ceph-mon[63900]: Deploying daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:23:13.005 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:12 smithi008 ceph-mon[63900]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T02:23:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:12 smithi077 ceph-mon[50325]: Upgrade: Updating mds.foofs.smithi008.wzyiku 2024-04-27T02:23:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:12 smithi077 ceph-mon[50325]: Deploying daemon mds.foofs.smithi008.wzyiku on smithi008 2024-04-27T02:23:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:12 smithi077 ceph-mon[50325]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T02:23:14.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:13 smithi077 ceph-mon[50325]: pgmap v312: 97 pgs: 6 peering, 14 active+undersized, 22 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.7 KiB/s wr, 9 op/s; 65/426 objects degraded (15.258%) 2024-04-27T02:23:14.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:13 smithi077 ceph-mon[50325]: Health check update: Degraded data redundancy: 65/426 objects degraded (15.258%), 22 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:14.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:13 smithi077 ceph-mon[50325]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T02:23:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:13 smithi008 ceph-mon[63900]: pgmap v312: 97 pgs: 6 peering, 14 active+undersized, 22 active+undersized+degraded, 55 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.7 KiB/s wr, 9 op/s; 65/426 objects degraded (15.258%) 2024-04-27T02:23:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:13 smithi008 ceph-mon[63900]: Health check update: Degraded data redundancy: 65/426 objects degraded (15.258%), 22 pgs degraded (PG_DEGRADED) 2024-04-27T02:23:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:13 smithi008 ceph-mon[63900]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T02:23:14.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[63896]: 2024-04-27T02:23:14.302+0000 7f1f8207e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: Standby daemon mds.foofs.smithi077.qcnfuu assigned to filesystem foofs as rank 0 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:replay} 2024-04-27T02:23:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:14 smithi008 ceph-mon[63900]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: Standby daemon mds.foofs.smithi077.qcnfuu assigned to filesystem foofs as rank 0 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T02:23:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:23:15.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T02:23:15.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:replay} 2024-04-27T02:23:15.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:14 smithi077 ceph-mon[50325]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T02:23:16.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:15 smithi008 ceph-mon[63900]: pgmap v316: 97 pgs: 6 peering, 14 active+undersized, 21 active+undersized+degraded, 56 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 57/426 objects degraded (13.380%) 2024-04-27T02:23:16.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:15 smithi008 ceph-mon[63900]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T02:23:16.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:15 smithi077 ceph-mon[50325]: pgmap v316: 97 pgs: 6 peering, 14 active+undersized, 21 active+undersized+degraded, 56 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 57/426 objects degraded (13.380%) 2024-04-27T02:23:16.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:15 smithi077 ceph-mon[50325]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T02:23:17.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:17.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:17 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:17.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:17 smithi077 ceph-mon[50325]: pgmap v318: 97 pgs: 6 peering, 91 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 511 B/s wr, 8 op/s; 1.6 MiB/s, 71 objects/s recovering 2024-04-27T02:23:17.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:17 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:reconnect 2024-04-27T02:23:17.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:17 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:reconnect} 2024-04-27T02:23:17.620 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:17.620 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:17 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:17.620 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:17 smithi008 ceph-mon[63900]: pgmap v318: 97 pgs: 6 peering, 91 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 511 B/s wr, 8 op/s; 1.6 MiB/s, 71 objects/s recovering 2024-04-27T02:23:17.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:17 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:reconnect 2024-04-27T02:23:17.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:17 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:reconnect} 2024-04-27T02:23:18.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: reconnect by client.24361 172.21.15.77:0/3491732489 after 0.00099998 2024-04-27T02:23:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: reconnect by client.14542 172.21.15.8:0/2391753012 after 0.00099998 2024-04-27T02:23:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/426 objects degraded (13.380%), 21 pgs degraded) 2024-04-27T02:23:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:rejoin 2024-04-27T02:23:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:rejoin} 2024-04-27T02:23:18.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:18 smithi077 ceph-mon[50325]: daemon mds.foofs.smithi077.qcnfuu is now active in filesystem foofs as rank 0 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: reconnect by client.24361 172.21.15.77:0/3491732489 after 0.00099998 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: reconnect by client.14542 172.21.15.8:0/2391753012 after 0.00099998 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/426 objects degraded (13.380%), 21 pgs degraded) 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:rejoin 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi077.qcnfuu=up:rejoin} 2024-04-27T02:23:18.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:18 smithi008 ceph-mon[63900]: daemon mds.foofs.smithi077.qcnfuu is now active in filesystem foofs as rank 0 2024-04-27T02:23:19.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:19 smithi077 ceph-mon[50325]: pgmap v319: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 391 B/s wr, 10 op/s; 1.2 MiB/s, 61 objects/s recovering 2024-04-27T02:23:19.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:19 smithi077 ceph-mon[50325]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:23:19.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:19 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:active 2024-04-27T02:23:19.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:19 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi077.qcnfuu=up:active} 2024-04-27T02:23:19.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:19 smithi008 ceph-mon[63900]: pgmap v319: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 391 B/s wr, 10 op/s; 1.2 MiB/s, 61 objects/s recovering 2024-04-27T02:23:19.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:19 smithi008 ceph-mon[63900]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:23:19.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:19 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.77:6832/4030729448,v1:172.21.15.77:6833/4030729448] up:active 2024-04-27T02:23:19.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:19 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi077.qcnfuu=up:active} 2024-04-27T02:23:21.744 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:21 smithi008 ceph-mon[63900]: pgmap v320: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 650 B/s wr, 15 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-04-27T02:23:21.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:21 smithi077 ceph-mon[50325]: pgmap v320: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 650 B/s wr, 15 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:boot 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi077.qcnfuu=up:active} 1 up:standby 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:22.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:22 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T02:23:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:boot 2024-04-27T02:23:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi077.qcnfuu=up:active} 1 up:standby 2024-04-27T02:23:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:23:22.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:22.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:22 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:23.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:23 smithi008 ceph-mon[63900]: pgmap v321: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 507 B/s wr, 11 op/s; 814 KiB/s, 40 objects/s recovering 2024-04-27T02:23:23.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:23 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:23.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:23 smithi077 ceph-mon[50325]: pgmap v321: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 507 B/s wr, 11 op/s; 814 KiB/s, 40 objects/s recovering 2024-04-27T02:23:23.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:23 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:25.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:25 smithi008 ceph-mon[63900]: pgmap v322: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 10 op/s; 656 KiB/s, 32 objects/s recovering 2024-04-27T02:23:25.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:25 smithi077 ceph-mon[50325]: pgmap v322: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 10 op/s; 656 KiB/s, 32 objects/s recovering 2024-04-27T02:23:27.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:27 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:27.832 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:27 smithi008 ceph-mon[63900]: pgmap v323: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 8.1 KiB/s wr, 11 op/s; 593 KiB/s, 3 objects/s recovering 2024-04-27T02:23:27.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:27 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:27.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:27 smithi077 ceph-mon[50325]: pgmap v323: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 8.1 KiB/s wr, 11 op/s; 593 KiB/s, 3 objects/s recovering 2024-04-27T02:23:29.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:29 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:29.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:29 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:29.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:29 smithi008 ceph-mon[63900]: pgmap v324: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.3 KiB/s wr, 9 op/s; 0 B/s, 3 objects/s recovering 2024-04-27T02:23:29.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:29 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:29.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:29 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:29.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:29 smithi077 ceph-mon[50325]: pgmap v324: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.3 KiB/s wr, 9 op/s; 0 B/s, 3 objects/s recovering 2024-04-27T02:23:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:30.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:30.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:31 smithi008 ceph-mon[63900]: pgmap v325: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-27T02:23:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:31 smithi077 ceph-mon[50325]: pgmap v325: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-27T02:23:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:31.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:33.365 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:23:33.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:33 smithi008 ceph-mon[63900]: pgmap v326: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-04-27T02:23:33.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:33 smithi077 ceph-mon[50325]: pgmap v326: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.2 KiB/s wr, 4 op/s 2024-04-27T02:23:34.063 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:23:34.063 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (10m) 5s ago 19m 24.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:23:34.063 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 5s ago 19m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:23:34.063 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (6m) 30s ago 18m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:23:34.063 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 5s ago 18m - - 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (11s) 5s ago 13m 13.7M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (13m) 30s ago 13m 15.3M - 17.2.0 e1d6a67b021e a6530484aa30 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (9m) 5s ago 20m 512M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (8m) 30s ago 18m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (7m) 5s ago 20m 63.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:23:34.064 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (7m) 30s ago 18m 45.1M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (12m) 5s ago 12m 281M - 4.0 e1d6a67b021e c850dc53ce63 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (12m) 30s ago 12m 88.9M - 4.0 e1d6a67b021e 593777b796b2 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (9m) 5s ago 18m 19.8M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (9m) 30s ago 18m 20.4M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (5m) 5s ago 17m 427M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (4m) 5s ago 17m 394M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 5s ago 16m 374M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:23:34.065 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 5s ago 16m 415M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:23:34.066 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (2m) 30s ago 15m 334M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:23:34.066 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (110s) 30s ago 15m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:23:34.066 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (72s) 30s ago 15m 275M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:23:34.066 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (35s) 30s ago 14m 15.1M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:23:34.066 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 5s ago 17m 70.4M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:23:34.487 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:23:34.487 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:23:34.487 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:23:34.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T02:23:34.488 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 13 2024-04-27T02:23:34.489 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:23:34.489 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:23:34.840 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:23:34.841 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "15/23 daemons upgraded", 2024-04-27T02:23:34.841 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading mds daemons", 2024-04-27T02:23:34.841 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:23:34.841 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.841 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi077.qcnfuu"]}]: dispatch 2024-04-27T02:23:34.842 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:34 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2101962968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:34.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi077.qcnfuu"]}]: dispatch 2024-04-27T02:23:34.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:34 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2101962968' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:35.259 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:23:35.259 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:23:35.259 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:23:35.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: Detected new or changed devices on smithi008 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: pgmap v327: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.5 KiB/s wr, 4 op/s 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: pgmap v328: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: Upgrade: It appears safe to stop mds.foofs.smithi077.qcnfuu 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:35.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:35 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/649178608' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:23:35.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: Detected new or changed devices on smithi008 2024-04-27T02:23:35.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: pgmap v327: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.5 KiB/s wr, 4 op/s 2024-04-27T02:23:35.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: pgmap v328: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:23:35.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:35.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: Upgrade: It appears safe to stop mds.foofs.smithi077.qcnfuu 2024-04-27T02:23:35.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:23:35.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:35 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/649178608' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:23:37.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: pgmap v329: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 217 B/s wr, 0 op/s 2024-04-27T02:23:37.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: Upgrade: Updating mds.foofs.smithi077.qcnfuu 2024-04-27T02:23:37.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:37.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:23:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:37.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:37 smithi077 ceph-mon[50325]: Deploying daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:23:37.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: pgmap v329: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 217 B/s wr, 0 op/s 2024-04-27T02:23:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: Upgrade: Updating mds.foofs.smithi077.qcnfuu 2024-04-27T02:23:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi077.qcnfuu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T02:23:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:37 smithi008 ceph-mon[63900]: Deploying daemon mds.foofs.smithi077.qcnfuu on smithi077 2024-04-27T02:23:38.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:38 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[63896]: 2024-04-27T02:23:38.208+0000 7f1f8207e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:39.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: pgmap v330: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 217 B/s wr, 0 op/s 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: Standby daemon mds.foofs.smithi008.wzyiku assigned to filesystem foofs as rank 0 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T02:23:39.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:39 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:replay} 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: pgmap v330: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 217 B/s wr, 0 op/s 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: Standby daemon mds.foofs.smithi008.wzyiku assigned to filesystem foofs as rank 0 2024-04-27T02:23:39.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T02:23:39.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T02:23:39.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T02:23:39.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:39 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:replay} 2024-04-27T02:23:41.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:41 smithi008 ceph-mon[63900]: pgmap v332: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 275 B/s wr, 4 op/s 2024-04-27T02:23:41.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:41 smithi008 ceph-mon[63900]: reconnect by client.14542 172.21.15.8:0/2391753012 after 0 2024-04-27T02:23:41.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:41 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:reconnect 2024-04-27T02:23:41.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:41 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:reconnect} 2024-04-27T02:23:41.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:41 smithi008 ceph-mon[63900]: reconnect by client.24361 172.21.15.77:0/3491732489 after 0.0529995 2024-04-27T02:23:41.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:41 smithi077 ceph-mon[50325]: pgmap v332: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 275 B/s wr, 4 op/s 2024-04-27T02:23:41.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:41 smithi077 ceph-mon[50325]: reconnect by client.14542 172.21.15.8:0/2391753012 after 0 2024-04-27T02:23:41.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:41 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:reconnect 2024-04-27T02:23:41.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:41 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:reconnect} 2024-04-27T02:23:41.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:41 smithi077 ceph-mon[50325]: reconnect by client.24361 172.21.15.77:0/3491732489 after 0.0529995 2024-04-27T02:23:42.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:42 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:rejoin 2024-04-27T02:23:42.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:42 smithi008 ceph-mon[63900]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:rejoin} 2024-04-27T02:23:42.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:42 smithi008 ceph-mon[63900]: daemon mds.foofs.smithi008.wzyiku is now active in filesystem foofs as rank 0 2024-04-27T02:23:42.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:42 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:rejoin 2024-04-27T02:23:42.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:42 smithi077 ceph-mon[50325]: fsmap foofs:1/1 {0=foofs.smithi008.wzyiku=up:rejoin} 2024-04-27T02:23:42.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:42 smithi077 ceph-mon[50325]: daemon mds.foofs.smithi008.wzyiku is now active in filesystem foofs as rank 0 2024-04-27T02:23:43.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:43 smithi008 ceph-mon[63900]: pgmap v333: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 255 B/s wr, 6 op/s 2024-04-27T02:23:43.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:43 smithi008 ceph-mon[63900]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:23:43.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:43 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:active 2024-04-27T02:23:43.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:43 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 2024-04-27T02:23:43.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:43 smithi077 ceph-mon[50325]: pgmap v333: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 255 B/s wr, 6 op/s 2024-04-27T02:23:43.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:43 smithi077 ceph-mon[50325]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T02:23:43.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:43 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877] up:active 2024-04-27T02:23:43.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:43 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 2024-04-27T02:23:45.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:45 smithi008 ceph-mon[63900]: pgmap v334: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 8 op/s 2024-04-27T02:23:45.422 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:45 smithi077 ceph-mon[50325]: pgmap v334: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 8 op/s 2024-04-27T02:23:46.833 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:46.833 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: mds.? [v2:172.21.15.77:6832/60930548,v1:172.21.15.77:6833/60930548] up:boot 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:23:46.834 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:47.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: mds.? [v2:172.21.15.77:6832/60930548,v1:172.21.15.77:6833/60930548] up:boot 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: fsmap foofs:1 {0=foofs.smithi008.wzyiku=up:active} 1 up:standby 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:23:47.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:23:47.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:47 smithi008 ceph-mon[63900]: pgmap v335: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-27T02:23:47.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:47 smithi077 ceph-mon[50325]: pgmap v335: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-27T02:23:50.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:49 smithi008 ceph-mon[63900]: pgmap v336: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-27T02:23:50.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:50.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:49 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:50.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:49 smithi077 ceph-mon[50325]: pgmap v336: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-27T02:23:50.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:50.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:49 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:51.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:51.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:50 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:51.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:50 smithi077 ceph-mon[50325]: pgmap v337: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 173 B/s wr, 9 op/s 2024-04-27T02:23:51.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:51.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:50 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:51.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:50 smithi008 ceph-mon[63900]: pgmap v337: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 173 B/s wr, 9 op/s 2024-04-27T02:23:53.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:53 smithi077 ceph-mon[50325]: pgmap v338: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-04-27T02:23:53.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:53 smithi008 ceph-mon[63900]: pgmap v338: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 7.3 KiB/s wr, 9 op/s 2024-04-27T02:23:54.503 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Detected new or changed devices on smithi077 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: pgmap v339: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 7.7 KiB/s wr, 8 op/s 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: pgmap v340: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 9.3 KiB/s wr, 6 op/s 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.504 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all mds 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wzyiku"}]': finished 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi077.qcnfuu"}]': finished 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all rgw 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all rbd-mirror 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.505 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all ceph-exporter 2024-04-27T02:23:54.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all iscsi 2024-04-27T02:23:54.506 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Detected new or changed devices on smithi077 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: pgmap v339: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 7.7 KiB/s wr, 8 op/s 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: pgmap v340: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 9.3 KiB/s wr, 6 op/s 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all mds 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wzyiku"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi008.wzyiku"}]': finished 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi077.qcnfuu"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi077.qcnfuu"}]': finished 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all rgw 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all rbd-mirror 2024-04-27T02:23:54.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all ceph-exporter 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all iscsi 2024-04-27T02:23:54.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:56.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: pgmap v341: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:23:56.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Upgrade: Updating nfs.foo.0.0.smithi008.lcixut (1/2) 2024-04-27T02:23:56.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:56.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Creating key for client.nfs.foo.0.0.smithi008.lcixut 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Rados config object exists: conf-nfs.foo 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Creating key for client.nfs.foo.0.0.smithi008.lcixut-rgw 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Bind address in nfs.foo.0.0.smithi008.lcixut's ganesha conf is defaulting to empty 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:56.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:56 smithi077 ceph-mon[50325]: Deploying daemon nfs.foo.0.0.smithi008.lcixut on smithi008 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: pgmap v341: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 217 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Upgrade: Updating nfs.foo.0.0.smithi008.lcixut (1/2) 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Creating key for client.nfs.foo.0.0.smithi008.lcixut 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:56.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Rados config object exists: conf-nfs.foo 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Creating key for client.nfs.foo.0.0.smithi008.lcixut-rgw 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi008.lcixut-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Bind address in nfs.foo.0.0.smithi008.lcixut's ganesha conf is defaulting to empty 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:23:56.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:56 smithi008 ceph-mon[63900]: Deploying daemon nfs.foo.0.0.smithi008.lcixut on smithi008 2024-04-27T02:23:58.487 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:23:58 smithi008 ceph-mon[63900]: pgmap v342: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 435 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:23:58.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:23:58 smithi077 ceph-mon[50325]: pgmap v342: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 435 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:24:00.710 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:00 smithi077 ceph-mon[50325]: pgmap v343: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 435 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:24:00.710 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:00 smithi077 ceph-mon[50325]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2391753012"}]: dispatch 2024-04-27T02:24:00.710 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:00 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:00.710 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:00 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:00.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:00 smithi008 ceph-mon[63900]: pgmap v343: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 435 B/s rd, 9.1 KiB/s wr, 4 op/s 2024-04-27T02:24:00.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:00 smithi008 ceph-mon[63900]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2391753012"}]: dispatch 2024-04-27T02:24:00.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:00 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:00.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:00 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:01 smithi077 ceph-mon[50325]: Evicting (and blocklisting) client session 14542 (172.21.15.8:0/2391753012) 2024-04-27T02:24:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:01 smithi077 ceph-mon[50325]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2391753012"}]': finished 2024-04-27T02:24:01.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:01 smithi077 ceph-mon[50325]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T02:24:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:01.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:01 smithi008 ceph-mon[63900]: Evicting (and blocklisting) client session 14542 (172.21.15.8:0/2391753012) 2024-04-27T02:24:01.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:01 smithi008 ceph-mon[63900]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.8:0/2391753012"}]': finished 2024-04-27T02:24:01.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:01 smithi008 ceph-mon[63900]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T02:24:01.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: pgmap v345: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 1 op/s 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Upgrade: Updating nfs.foo.1.0.smithi077.lnlzww (2/2) 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:02.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Rados config object exists: conf-nfs.foo 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww-rgw 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Bind address in nfs.foo.1.0.smithi077.lnlzww's ganesha conf is defaulting to empty 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:02.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:02 smithi077 ceph-mon[50325]: Deploying daemon nfs.foo.1.0.smithi077.lnlzww on smithi077 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: pgmap v345: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 1 op/s 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Upgrade: Updating nfs.foo.1.0.smithi077.lnlzww (2/2) 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T02:24:02.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Rados config object exists: conf-nfs.foo 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Creating key for client.nfs.foo.1.0.smithi077.lnlzww-rgw 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi077.lnlzww-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Bind address in nfs.foo.1.0.smithi077.lnlzww's ganesha conf is defaulting to empty 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:02.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:02 smithi008 ceph-mon[63900]: Deploying daemon nfs.foo.1.0.smithi077.lnlzww on smithi077 2024-04-27T02:24:04.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:04 smithi077 ceph-mon[50325]: pgmap v346: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T02:24:04.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:04 smithi008 ceph-mon[63900]: pgmap v346: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T02:24:05.652 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (10m) 38s ago 19m 24.2M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (6m) 38s ago 19m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (6m) 17s ago 18m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 38s ago 19m - - 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (44s) 38s ago 13m 13.7M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (21s) 17s ago 13m 13.7M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3f47f10d2526 2024-04-27T02:24:06.398 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (9m) 38s ago 20m 512M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (8m) 17s ago 18m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 38s ago 21m 63.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (7m) 17s ago 18m 46.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 starting - - - - 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 starting - - - - 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (10m) 38s ago 19m 19.8M - 1.3.1 1dbe0e931976 bd0075666671 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (9m) 17s ago 18m 17.3M - 1.3.1 1dbe0e931976 98b23cd68663 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 38s ago 18m 427M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 38s ago 17m 394M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (4m) 38s ago 17m 374M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (3m) 38s ago 16m 415M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (3m) 17s ago 16m 337M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (2m) 17s ago 15m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (104s) 17s ago 15m 282M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (67s) 17s ago 15m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:24:06.399 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (8m) 38s ago 18m 70.4M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:24:06.403 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:06.403 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:06.403 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:06 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:06.404 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:06 smithi008 ceph-mon[63900]: pgmap v347: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-27T02:24:06.404 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:06 smithi008 ceph-mon[63900]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:06.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:06.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:06.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:06 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:06.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:06 smithi077 ceph-mon[50325]: pgmap v347: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-27T02:24:06.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:06 smithi077 ceph-mon[50325]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:06.830 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:24:06.830 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:24:06.830 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:24:06.831 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:24:07.201 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:24:07.201 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "16/23 daemons upgraded", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading nfs daemons", 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:24:07.202 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:24:07.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:07 smithi008 ceph-mon[63900]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:07.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:07 smithi008 ceph-mon[63900]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:07.621 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:07 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3253520332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:07.659 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:24:07.660 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:24:07.660 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:24:07.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:07 smithi077 ceph-mon[50325]: from='client.44405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:07.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:07 smithi077 ceph-mon[50325]: from='client.34526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:07.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:07 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3253520332' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:08.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:08 smithi008 ceph-mon[63900]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:08.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:08 smithi008 ceph-mon[63900]: pgmap v348: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-27T02:24:08.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:08 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/132834525' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:24:08.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:08 smithi077 ceph-mon[50325]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:08.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:08 smithi077 ceph-mon[50325]: pgmap v348: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-27T02:24:08.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:08 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/132834525' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:24:09.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:09 smithi008 ceph-mon[63900]: Evicting (and blocklisting) client session 24361 (172.21.15.77:0/3491732489) 2024-04-27T02:24:09.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:09 smithi008 ceph-mon[63900]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.77:0/3491732489"}]: dispatch 2024-04-27T02:24:09.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:09.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:09.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:09 smithi077 ceph-mon[50325]: Evicting (and blocklisting) client session 24361 (172.21.15.77:0/3491732489) 2024-04-27T02:24:09.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:09 smithi077 ceph-mon[50325]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.77:0/3491732489"}]: dispatch 2024-04-27T02:24:09.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:09.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.77:0/3491732489"}]': finished 2024-04-27T02:24:10.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: osdmap e156: 8 total, 8 up, 8 in 2024-04-27T02:24:10.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: pgmap v350: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-04-27T02:24:10.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.658 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: from='mds.? [v2:172.21.15.8:6834/2324451877,v1:172.21.15.8:6835/2324451877]' entity='mds.foofs.smithi008.wzyiku' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.77:0/3491732489"}]': finished 2024-04-27T02:24:10.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: osdmap e156: 8 total, 8 up, 8 in 2024-04-27T02:24:10.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: pgmap v350: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-04-27T02:24:10.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:10.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:12.586 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:12 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:12.587 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:12 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:12.587 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:12 smithi077 ceph-mon[50325]: pgmap v351: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-27T02:24:12.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:12.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:12 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:12.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:12 smithi008 ceph-mon[63900]: pgmap v351: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-27T02:24:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:14 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:14 smithi008 ceph-mon[63900]: pgmap v352: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T02:24:14.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:14.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:14 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:14.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:14 smithi077 ceph-mon[50325]: pgmap v352: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: pgmap v353: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: pgmap v354: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 128 B/s rd, 514 B/s wr, 1 op/s 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:16.394 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all nfs 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.lcixut"}]: dispatch 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.lcixut"}]': finished 2024-04-27T02:24:16.395 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi077.lnlzww"}]: dispatch 2024-04-27T02:24:16.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi077.lnlzww"}]': finished 2024-04-27T02:24:16.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: Upgrade: Setting container_image for all nvmeof 2024-04-27T02:24:16.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.396 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: pgmap v353: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: pgmap v354: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 128 B/s rd, 514 B/s wr, 1 op/s 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all nfs 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.lcixut"}]: dispatch 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi008.lcixut"}]': finished 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi077.lnlzww"}]: dispatch 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi077.lnlzww"}]': finished 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: Upgrade: Setting container_image for all nvmeof 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:16.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:18.654 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:18 smithi008 ceph-mon[63900]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-04-27T02:24:18.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:18 smithi008 ceph-mon[63900]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-27T02:24:18.655 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:18 smithi008 ceph-mon[63900]: pgmap v355: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 509 B/s wr, 0 op/s 2024-04-27T02:24:18.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:18 smithi077 ceph-mon[50325]: Upgrade: Updating node-exporter.smithi008 (1/2) 2024-04-27T02:24:18.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:18 smithi077 ceph-mon[50325]: Deploying daemon node-exporter.smithi008 on smithi008 2024-04-27T02:24:18.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:18 smithi077 ceph-mon[50325]: pgmap v355: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 127 B/s rd, 509 B/s wr, 0 op/s 2024-04-27T02:24:20.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:20 smithi008 ceph-mon[63900]: pgmap v356: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 411 B/s wr, 0 op/s 2024-04-27T02:24:20.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:20 smithi077 ceph-mon[50325]: pgmap v356: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 411 B/s wr, 0 op/s 2024-04-27T02:24:22.705 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:22 smithi008 ceph-mon[63900]: pgmap v357: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 0 op/s 2024-04-27T02:24:22.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:22 smithi077 ceph-mon[50325]: pgmap v357: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 0 B/s wr, 0 op/s 2024-04-27T02:24:24.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:23 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:24.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:23 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:24.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:23 smithi077 ceph-mon[50325]: pgmap v358: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2024-04-27T02:24:24.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:23 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:24.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:23 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:24.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:23 smithi008 ceph-mon[63900]: pgmap v358: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2024-04-27T02:24:25.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:25 smithi077 ceph-mon[50325]: Upgrade: Updating node-exporter.smithi077 (2/2) 2024-04-27T02:24:25.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:25 smithi077 ceph-mon[50325]: Deploying daemon node-exporter.smithi077 on smithi077 2024-04-27T02:24:25.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:25 smithi008 ceph-mon[63900]: Upgrade: Updating node-exporter.smithi077 (2/2) 2024-04-27T02:24:25.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:25 smithi008 ceph-mon[63900]: Deploying daemon node-exporter.smithi077 on smithi077 2024-04-27T02:24:26.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:26 smithi008 ceph-mon[63900]: pgmap v359: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:24:26.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:26 smithi077 ceph-mon[50325]: pgmap v359: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:24:28.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:28 smithi008 ceph-mon[63900]: pgmap v360: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:28.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:28 smithi077 ceph-mon[50325]: pgmap v360: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:30.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: pgmap v361: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:30.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:30.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: pgmap v361: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:30.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:31.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:32.881 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:32 smithi008 ceph-mon[63900]: pgmap v362: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:32.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:32 smithi077 ceph-mon[50325]: pgmap v362: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:33.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:33.862 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:33 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:33 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:34 smithi077 ceph-mon[50325]: pgmap v363: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:24:34.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:34 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:34 smithi008 ceph-mon[63900]: pgmap v363: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:24:34.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:34.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:34 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:36.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:36 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:36.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:36 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:36.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:36 smithi077 ceph-mon[50325]: pgmap v364: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:36.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:36 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:36.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:36 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:36.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:36 smithi008 ceph-mon[63900]: pgmap v364: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:37.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:37 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:37.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:37 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:37 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:37.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:37 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:38.034 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:24:38.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:38 smithi008 ceph-mon[63900]: pgmap v365: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:38.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:38 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:38.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:38 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (11m) 3s ago 20m 24.3M - 0.23.0 ba2b418f427c 0ade09fdfcd4 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 3s ago 20m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (7m) 5s ago 19m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 3s ago 19m - - 2024-04-27T02:24:38.725 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (76s) 3s ago 14m 31.8M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (53s) 5s ago 14m 13.9M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3f47f10d2526 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (10m) 3s ago 21m 514M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (9m) 5s ago 19m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (8m) 3s ago 21m 66.5M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (8m) 5s ago 19m 46.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (39s) 3s ago 13m 62.0M - 5.7 87e5280d5ae8 447d498cec7b 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (33s) 5s ago 13m 59.5M - 5.7 87e5280d5ae8 cc5286f086dc 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (16s) 3s ago 19m 8074k - 1.5.0 0da6a335fe13 edcd5f8c65c8 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (9s) 5s ago 19m 3363k - 1.5.0 0da6a335fe13 ed9e63d6a604 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (6m) 3s ago 18m 427M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (5m) 3s ago 18m 396M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 3s ago 17m 376M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 3s ago 17m 417M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:24:38.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (3m) 5s ago 16m 338M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:24:38.727 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (2m) 5s ago 16m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:24:38.727 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (2m) 5s ago 16m 283M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:24:38.727 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (99s) 5s ago 15m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:24:38.727 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (9m) 3s ago 18m 70.4M - 2.33.4 514e6a882f6e 13fe9be8b8b7 2024-04-27T02:24:38.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:38 smithi077 ceph-mon[50325]: pgmap v365: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:38.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:38 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:38.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:38 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:39.220 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:24:39.220 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:24:39.221 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:24:39.575 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:24:39.575 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:24:39.575 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:24:39.576 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:24:39.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:39 smithi077 ceph-mon[50325]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:39 smithi077 ceph-mon[50325]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:39 smithi077 ceph-mon[50325]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:39 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3396212918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:39.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:39 smithi008 ceph-mon[63900]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:39 smithi008 ceph-mon[63900]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:39 smithi008 ceph-mon[63900]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:39.763 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:39 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3396212918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.000 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:24:40.000 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:24:40.000 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: pgmap v366: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: pgmap v367: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1657147033' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:40 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: pgmap v366: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:24:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: pgmap v367: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1657147033' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:40.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:40 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:43.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:42 smithi008 ceph-mon[63900]: Upgrade: Updating prometheus.smithi008 2024-04-27T02:24:43.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:42 smithi008 ceph-mon[63900]: pgmap v368: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:43.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:42 smithi077 ceph-mon[50325]: Upgrade: Updating prometheus.smithi008 2024-04-27T02:24:43.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:42 smithi077 ceph-mon[50325]: pgmap v368: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:43.987 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:43 smithi008 ceph-mon[63900]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-27T02:24:44.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:43 smithi077 ceph-mon[50325]: Deploying daemon prometheus.smithi008 on smithi008 2024-04-27T02:24:45.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:44 smithi008 ceph-mon[63900]: pgmap v369: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:45.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:44 smithi077 ceph-mon[50325]: pgmap v369: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:47.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:46 smithi008 ceph-mon[63900]: pgmap v370: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:47.132 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:47.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:46 smithi077 ceph-mon[50325]: pgmap v370: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:47.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:24:48.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:48 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:48.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:48 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:48.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:48 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:48.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:48 smithi008 ceph-mon[63900]: pgmap v371: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:48.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:48 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:48.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:48 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:48.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:48 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:48.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:48 smithi077 ceph-mon[50325]: pgmap v371: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:51.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:50 smithi008 ceph-mon[63900]: pgmap v372: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:51.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:50 smithi077 ceph-mon[50325]: pgmap v372: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2024-04-27T02:24:53.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:52 smithi008 ceph-mon[63900]: pgmap v373: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:53.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:53.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:52 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:53.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:52 smithi077 ceph-mon[50325]: pgmap v373: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:53.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:53.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:52 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:54.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:54.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:54 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:54.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:54 smithi008 ceph-mon[63900]: pgmap v374: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:24:54.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:54.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:54 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:54.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:54 smithi077 ceph-mon[50325]: pgmap v374: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:24:57.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: pgmap v375: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:57.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:57.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:57.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:56 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: pgmap v375: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:57.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:56 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: pgmap v376: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: pgmap v377: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:57 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: pgmap v376: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: pgmap v377: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:58.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:57 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:24:59.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:58 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:24:59.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:58 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T02:25:00.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:59 smithi008 ceph-mon[63900]: Upgrade: Updating alertmanager.smithi008 2024-04-27T02:25:00.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:59 smithi008 ceph-mon[63900]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-27T02:25:00.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:24:59 smithi008 ceph-mon[63900]: pgmap v378: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:00.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:59 smithi077 ceph-mon[50325]: Upgrade: Updating alertmanager.smithi008 2024-04-27T02:25:00.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:59 smithi077 ceph-mon[50325]: Deploying daemon alertmanager.smithi008 on smithi008 2024-04-27T02:25:00.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:24:59 smithi077 ceph-mon[50325]: pgmap v378: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:02.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:01 smithi008 ceph-mon[63900]: pgmap v379: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:02.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:02.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:01 smithi077 ceph-mon[50325]: pgmap v379: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:02.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:03.925 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:03 smithi008 ceph-mon[63900]: pgmap v380: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:04.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:03 smithi077 ceph-mon[50325]: pgmap v380: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:05.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:05.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:05.406 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:05 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:05.407 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:05 smithi008 ceph-mon[63900]: pgmap v381: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:05.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:05.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:05.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:05 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:05.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:05 smithi077 ceph-mon[50325]: pgmap v381: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2024-04-27T02:25:07.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:06 smithi077 ceph-mon[50325]: pgmap v382: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:25:07.330 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:06 smithi008 ceph-mon[63900]: pgmap v382: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:25:10.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:09 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:10.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:09 smithi008 ceph-mon[63900]: pgmap v383: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:10.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:10.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:09 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:10.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:09 smithi077 ceph-mon[50325]: pgmap v383: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:10.365 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:25:10.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:10.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:10 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (7s) 2s ago 20m 16.1M - 0.25.0 c8568f914cd2 d86462087031 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (7m) 2s ago 20m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (7m) 38s ago 20m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 error 2s ago 20m - - 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (108s) 2s ago 14m 32.0M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (85s) 38s ago 14m 13.9M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3f47f10d2526 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (10m) 2s ago 21m 516M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (9m) 38s ago 19m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (9m) 2s ago 22m 68.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (8m) 38s ago 19m 46.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:25:11.067 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (71s) 2s ago 14m 62.4M - 5.7 87e5280d5ae8 447d498cec7b 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (66s) 38s ago 14m 59.5M - 5.7 87e5280d5ae8 cc5286f086dc 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (48s) 2s ago 20m 8530k - 1.5.0 0da6a335fe13 edcd5f8c65c8 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (41s) 38s ago 19m 3363k - 1.5.0 0da6a335fe13 ed9e63d6a604 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 2s ago 19m 428M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 2s ago 18m 398M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (5m) 2s ago 18m 377M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (4m) 2s ago 17m 417M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (4m) 38s ago 17m 338M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (3m) 38s ago 17m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (2m) 38s ago 16m 283M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (2m) 38s ago 16m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:25:11.068 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (24s) 2s ago 19m 48.8M - 2.43.0 a07b618ecd1d 952e6071f501 2024-04-27T02:25:11.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:11.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:10 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:25:11.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:25:11.487 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": true, 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [ 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "mon", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "crash", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "mds", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "nfs", 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr" 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:25:11.837 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": "18/23 daemons upgraded", 2024-04-27T02:25:11.838 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-27T02:25:11.838 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:25:11.838 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:25:12.057 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:11 smithi008 ceph-mon[63900]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:12.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:11 smithi008 ceph-mon[63900]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:12.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:11 smithi008 ceph-mon[63900]: pgmap v384: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:12.058 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:11 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/166885814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:12.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:11 smithi077 ceph-mon[50325]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:12.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:11 smithi077 ceph-mon[50325]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:12.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:11 smithi077 ceph-mon[50325]: pgmap v384: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:12.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:11 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/166885814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:12.251 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T02:25:12.251 INFO:teuthology.orchestra.run.smithi008.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T02:25:12.251 INFO:teuthology.orchestra.run.smithi008.stdout: daemon grafana.smithi008 on smithi008 is in error state 2024-04-27T02:25:13.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:12 smithi008 ceph-mon[63900]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:13.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:12 smithi008 ceph-mon[63900]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:13.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:12 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/1671316580' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:25:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:12 smithi077 ceph-mon[50325]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:12 smithi077 ceph-mon[50325]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:13.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:12 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/1671316580' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: pgmap v385: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008:9093"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:13 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: pgmap v385: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:14.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008:9093"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:14.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:13 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:15.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:14 smithi008 ceph-mon[63900]: pgmap v386: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-27T02:25:15.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:14 smithi008 ceph-mon[63900]: pgmap v387: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:14 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:25:15.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:14 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008:9093"}]: dispatch 2024-04-27T02:25:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:14 smithi077 ceph-mon[50325]: pgmap v386: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-27T02:25:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:14 smithi077 ceph-mon[50325]: pgmap v387: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:14 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T02:25:15.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:14 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008:9093"}]: dispatch 2024-04-27T02:25:17.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:16 smithi008 ceph-mon[63900]: Upgrade: Updating grafana.smithi008 2024-04-27T02:25:17.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:16 smithi008 ceph-mon[63900]: Deploying daemon grafana.smithi008 on smithi008 2024-04-27T02:25:17.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:16 smithi008 ceph-mon[63900]: pgmap v388: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:17.098 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:17.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:16 smithi077 ceph-mon[50325]: Upgrade: Updating grafana.smithi008 2024-04-27T02:25:17.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:16 smithi077 ceph-mon[50325]: Deploying daemon grafana.smithi008 on smithi008 2024-04-27T02:25:17.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:16 smithi077 ceph-mon[50325]: pgmap v388: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:17.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:19.180 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:18 smithi008 ceph-mon[63900]: pgmap v389: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:19.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:18 smithi077 ceph-mon[50325]: pgmap v389: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:20.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:20.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:20.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:20 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:20.657 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:20 smithi008 ceph-mon[63900]: pgmap v390: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:20.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:20.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:20.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:20 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:20.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:20 smithi077 ceph-mon[50325]: pgmap v390: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 234 B/s rd, 0 op/s 2024-04-27T02:25:22.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:22 smithi008 ceph-mon[63900]: pgmap v391: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:25:22.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:22 smithi077 ceph-mon[50325]: pgmap v391: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T02:25:24.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:24 smithi008 ceph-mon[63900]: pgmap v392: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:25:24.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:24 smithi077 ceph-mon[50325]: pgmap v392: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T02:25:26.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:26.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:25 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:26.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:26.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:25 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:26.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:26 smithi008 ceph-mon[63900]: pgmap v393: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:26.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:26 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:26.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:26 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:27.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:26 smithi077 ceph-mon[50325]: pgmap v393: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:27.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:26 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:27.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:26 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:29.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:28 smithi008 ceph-mon[63900]: pgmap v394: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:29.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:28 smithi077 ceph-mon[50325]: pgmap v394: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: pgmap v395: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008:3000"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008:3000"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.908 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.909 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: Upgrade: Finalizing container_image settings 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-27T02:25:30.910 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.911 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: Upgrade: Complete! 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.912 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.913 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: pgmap v395: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008:3000"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi008:3000"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: Upgrade: Finalizing container_image settings 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-27T02:25:30.985 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.986 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: Upgrade: Complete! 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:30.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:31.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:31 smithi008 ceph-mon[63900]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:25:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:31 smithi008 ceph-mon[63900]: Cluster is now healthy 2024-04-27T02:25:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:31.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:31 smithi077 ceph-mon[50325]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T02:25:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:31 smithi077 ceph-mon[50325]: Cluster is now healthy 2024-04-27T02:25:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:32.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:32 smithi008 ceph-mon[63900]: pgmap v396: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:32.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:32 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:32.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:32 smithi077 ceph-mon[50325]: pgmap v396: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:32.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:32 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:25:34.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:34 smithi008 ceph-mon[63900]: pgmap v397: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2024-04-27T02:25:34.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:34 smithi077 ceph-mon[50325]: pgmap v397: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2024-04-27T02:25:36.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:36 smithi008 ceph-mon[63900]: pgmap v398: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:36.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:36 smithi077 ceph-mon[50325]: pgmap v398: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:38.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:38 smithi008 ceph-mon[63900]: pgmap v399: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:38 smithi077 ceph-mon[50325]: pgmap v399: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:40.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:40 smithi008 ceph-mon[63900]: pgmap v400: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:40 smithi077 ceph-mon[50325]: pgmap v400: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2024-04-27T02:25:42.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:42 smithi008 ceph-mon[63900]: pgmap v401: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:42.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:42 smithi077 ceph-mon[50325]: pgmap v401: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:43.383 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps' 2024-04-27T02:25:43.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:43 smithi008 ceph-mon[63900]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:43.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:43 smithi077 ceph-mon[50325]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:44.656 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:44 smithi008 ceph-mon[63900]: pgmap v402: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:44.883 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:44 smithi077 ceph-mon[50325]: pgmap v402: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:45.400 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:25:45.400 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (41s) 20s ago 21m 16.1M - 0.25.0 c8568f914cd2 d86462087031 2024-04-27T02:25:45.400 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (8m) 20s ago 21m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:25:45.400 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (8m) 72s ago 20m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:25:45.400 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (26s) 20s ago 21m 51.5M - 9.4.12 2bacad6d85d8 e1738ccf106f 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (2m) 20s ago 15m 32.1M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (2m) 72s ago 15m 13.9M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3f47f10d2526 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (11m) 20s ago 22m 522M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (10m) 72s ago 20m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (10m) 20s ago 22m 68.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (9m) 72s ago 20m 46.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (106s) 20s ago 15m 62.5M - 5.7 87e5280d5ae8 447d498cec7b 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (100s) 72s ago 14m 59.5M - 5.7 87e5280d5ae8 cc5286f086dc 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (82s) 20s ago 20m 7960k - 1.5.0 0da6a335fe13 edcd5f8c65c8 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (76s) 72s ago 20m 3363k - 1.5.0 0da6a335fe13 ed9e63d6a604 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (7m) 20s ago 19m 429M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (6m) 20s ago 19m 397M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (6m) 20s ago 18m 378M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (5m) 20s ago 18m 418M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:25:45.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (4m) 72s ago 18m 338M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:25:45.402 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (4m) 72s ago 17m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:25:45.402 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (3m) 72s ago 17m 283M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:25:45.402 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (2m) 72s ago 16m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:25:45.402 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (58s) 20s ago 20m 51.1M - 2.43.0 a07b618ecd1d 952e6071f501 2024-04-27T02:25:46.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions' 2024-04-27T02:25:46.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:46 smithi008 ceph-mon[63900]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:46.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:46 smithi008 ceph-mon[63900]: pgmap v403: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:46.907 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:46 smithi077 ceph-mon[50325]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:25:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:46 smithi077 ceph-mon[50325]: pgmap v403: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:25:48.044 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:25:48.045 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:25:48.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:25:48.046 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:25:48.046 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T02:25:48.046 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:25:48.046 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:25:48.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:48 smithi008 ceph-mon[63900]: pgmap v404: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:48.777 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:48 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3171597403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:48.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-27T02:25:48.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:48 smithi077 ceph-mon[50325]: pgmap v404: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:48.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:48 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3171597403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:25:50.630 INFO:teuthology.orchestra.run.smithi008.stdout:wait for servicemap items w/ changing names to refresh 2024-04-27T02:25:50.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:50 smithi008 ceph-mon[63900]: pgmap v405: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:50.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:50 smithi077 ceph-mon[50325]: pgmap v405: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:51.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'sleep 60' 2024-04-27T02:25:52.716 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:52 smithi008 ceph-mon[63900]: pgmap v406: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:52.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:52 smithi077 ceph-mon[50325]: pgmap v406: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:54.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:54 smithi008 ceph-mon[63900]: pgmap v407: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:54.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:54 smithi077 ceph-mon[50325]: pgmap v407: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:25:56.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:56 smithi008 ceph-mon[63900]: pgmap v408: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:56.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:56 smithi077 ceph-mon[50325]: pgmap v408: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:58.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:25:58 smithi008 ceph-mon[63900]: pgmap v409: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:25:58.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:25:58 smithi077 ceph-mon[50325]: pgmap v409: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:00.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:00 smithi008 ceph-mon[63900]: pgmap v410: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:00.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:00 smithi077 ceph-mon[50325]: pgmap v410: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:01.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:01.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:02.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:02 smithi008 ceph-mon[63900]: pgmap v411: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:02.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:02 smithi077 ceph-mon[50325]: pgmap v411: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:04.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:04 smithi008 ceph-mon[63900]: pgmap v412: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:04.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:04 smithi077 ceph-mon[50325]: pgmap v412: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:06.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:06 smithi008 ceph-mon[63900]: pgmap v413: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:06.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:06 smithi077 ceph-mon[50325]: pgmap v413: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:08 smithi008 ceph-mon[63900]: pgmap v414: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:08.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:08 smithi077 ceph-mon[50325]: pgmap v414: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:10.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:10 smithi008 ceph-mon[63900]: pgmap v415: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:10.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:10 smithi077 ceph-mon[50325]: pgmap v415: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:12.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:12 smithi008 ceph-mon[63900]: pgmap v416: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:12.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:12 smithi077 ceph-mon[50325]: pgmap v416: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:14.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:14 smithi008 ceph-mon[63900]: pgmap v417: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:14.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:14 smithi077 ceph-mon[50325]: pgmap v417: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:16.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:16 smithi008 ceph-mon[63900]: pgmap v418: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:16.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:16.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:16 smithi077 ceph-mon[50325]: pgmap v418: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:16.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:18.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:18 smithi077 ceph-mon[50325]: pgmap v419: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:19.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:18 smithi008 ceph-mon[63900]: pgmap v419: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:20.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:20 smithi077 ceph-mon[50325]: pgmap v420: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:21.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:20 smithi008 ceph-mon[63900]: pgmap v420: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:22.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:22 smithi077 ceph-mon[50325]: pgmap v421: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:23.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:22 smithi008 ceph-mon[63900]: pgmap v421: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:24.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:24 smithi077 ceph-mon[50325]: pgmap v422: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:25.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:24 smithi008 ceph-mon[63900]: pgmap v422: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:26.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:26 smithi077 ceph-mon[50325]: pgmap v423: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:27.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:26 smithi008 ceph-mon[63900]: pgmap v423: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:28.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:28 smithi077 ceph-mon[50325]: pgmap v424: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:29.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:28 smithi008 ceph-mon[63900]: pgmap v424: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:30.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:30 smithi077 ceph-mon[50325]: pgmap v425: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:30.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:30 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:26:31.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:30 smithi008 ceph-mon[63900]: pgmap v425: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:31.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:30 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T02:26:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:26:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:26:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:26:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:26:31.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:31 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:32.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T02:26:32.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T02:26:32.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:26:32.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' 2024-04-27T02:26:32.157 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:31 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:32.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:32 smithi077 ceph-mon[50325]: pgmap v426: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:33.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:32 smithi008 ceph-mon[63900]: pgmap v426: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:34.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:34 smithi077 ceph-mon[50325]: pgmap v427: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:35.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:34 smithi008 ceph-mon[63900]: pgmap v427: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:36.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:36 smithi077 ceph-mon[50325]: pgmap v428: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:37.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:36 smithi008 ceph-mon[63900]: pgmap v428: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:38.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:38 smithi077 ceph-mon[50325]: pgmap v429: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:39.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:38 smithi008 ceph-mon[63900]: pgmap v429: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:40.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:40 smithi077 ceph-mon[50325]: pgmap v430: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:41.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:40 smithi008 ceph-mon[63900]: pgmap v430: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:42.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:42 smithi077 ceph-mon[50325]: pgmap v431: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:43.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:42 smithi008 ceph-mon[63900]: pgmap v431: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:44.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:44 smithi077 ceph-mon[50325]: pgmap v432: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:45.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:44 smithi008 ceph-mon[63900]: pgmap v432: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:46 smithi077 ceph-mon[50325]: pgmap v433: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:46.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:46 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:47.158 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:46 smithi008 ceph-mon[63900]: pgmap v433: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:47.159 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:46 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:26:48.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:48 smithi077 ceph-mon[50325]: pgmap v434: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:49.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:48 smithi008 ceph-mon[63900]: pgmap v434: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:50.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:50 smithi077 ceph-mon[50325]: pgmap v435: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:51.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:50 smithi008 ceph-mon[63900]: pgmap v435: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:52.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:52 smithi077 ceph-mon[50325]: pgmap v436: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:53.130 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:52 smithi008 ceph-mon[63900]: pgmap v436: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:53.428 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps' 2024-04-27T02:26:54.837 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:54 smithi008 ceph-mon[63900]: pgmap v437: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:54.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:54 smithi077 ceph-mon[50325]: pgmap v437: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:26:55.294 INFO:teuthology.orchestra.run.smithi008.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T02:26:55.294 INFO:teuthology.orchestra.run.smithi008.stdout:alertmanager.smithi008 smithi008 *:9093,9094 running (111s) 90s ago 22m 16.1M - 0.25.0 c8568f914cd2 d86462087031 2024-04-27T02:26:55.294 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi008 smithi008 running (9m) 90s ago 22m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3a604a217401 2024-04-27T02:26:55.294 INFO:teuthology.orchestra.run.smithi008.stdout:crash.smithi077 smithi077 running (9m) 2m ago 21m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c3b1e43ab995 2024-04-27T02:26:55.294 INFO:teuthology.orchestra.run.smithi008.stdout:grafana.smithi008 smithi008 *:3000 running (96s) 90s ago 22m 51.5M - 9.4.12 2bacad6d85d8 e1738ccf106f 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi008.wzyiku smithi008 running (3m) 90s ago 16m 32.1M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2e61fa354c13 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mds.foofs.smithi077.qcnfuu smithi077 running (3m) 2m ago 16m 13.9M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3f47f10d2526 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi008.exkhsc smithi008 *:8443,9283,8765 running (12m) 90s ago 23m 522M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 fc9727401cc2 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mgr.smithi077.snksrp smithi077 *:8443,9283,8765 running (11m) 2m ago 21m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 9872ba6fd60e 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi008 smithi008 running (11m) 90s ago 23m 68.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 50e74d82ab4d 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:mon.smithi077 smithi077 running (10m) 2m ago 21m 46.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 288f6e8c6e2e 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.0.0.smithi008.lcixut smithi008 *:2049 running (2m) 90s ago 16m 62.5M - 5.7 87e5280d5ae8 447d498cec7b 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:nfs.foo.1.0.smithi077.lnlzww smithi077 *:2049 running (2m) 2m ago 16m 59.5M - 5.7 87e5280d5ae8 cc5286f086dc 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi008 smithi008 *:9100 running (2m) 90s ago 22m 7960k - 1.5.0 0da6a335fe13 edcd5f8c65c8 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:node-exporter.smithi077 smithi077 *:9100 running (2m) 2m ago 21m 3363k - 1.5.0 0da6a335fe13 ed9e63d6a604 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 smithi008 running (8m) 90s ago 20m 429M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 f1128c0b1706 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 smithi008 running (8m) 90s ago 20m 397M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 623e09212f51 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 smithi008 running (7m) 90s ago 19m 378M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 35f27121d198 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 smithi008 running (6m) 90s ago 19m 418M 1937M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 00420044291b 2024-04-27T02:26:55.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 smithi077 running (5m) 2m ago 19m 338M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 5b45fe64c65b 2024-04-27T02:26:55.296 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 smithi077 running (5m) 2m ago 18m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8ae0b8667e3 2024-04-27T02:26:55.296 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 smithi077 running (4m) 2m ago 18m 283M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d80d5951ed26 2024-04-27T02:26:55.296 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 smithi077 running (3m) 2m ago 17m 295M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 0baf0f4a3a03 2024-04-27T02:26:55.296 INFO:teuthology.orchestra.run.smithi008.stdout:prometheus.smithi008 smithi008 *:9095 running (2m) 90s ago 21m 51.1M - 2.43.0 a07b618ecd1d 952e6071f501 2024-04-27T02:26:55.960 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch upgrade status' 2024-04-27T02:26:56.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:56 smithi008 ceph-mon[63900]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:26:56.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:56 smithi008 ceph-mon[63900]: pgmap v438: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:56.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:56 smithi077 ceph-mon[50325]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:26:56.982 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:56 smithi077 ceph-mon[50325]: pgmap v438: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "target_image": null, 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "in_progress": false, 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "which": "", 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "services_complete": [], 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "progress": null, 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "message": "", 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout: "is_paused": false 2024-04-27T02:26:57.950 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:26:58.683 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph health detail' 2024-04-27T02:26:58.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:58 smithi008 ceph-mon[63900]: pgmap v439: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:58.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:58 smithi077 ceph-mon[50325]: pgmap v439: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:26:59.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:26:59 smithi077 ceph-mon[50325]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:26:59.996 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:26:59 smithi008 ceph-mon[63900]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:27:00.808 INFO:teuthology.orchestra.run.smithi008.stdout:HEALTH_OK 2024-04-27T02:27:00.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:00 smithi077 ceph-mon[50325]: pgmap v440: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:01.129 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:00 smithi008 ceph-mon[63900]: pgmap v440: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:01.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions' 2024-04-27T02:27:01.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:01 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/3782651175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:27:01.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:01 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:27:02.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:01 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/3782651175' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T02:27:02.007 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:01 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:27:03.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:02 smithi008 ceph-mon[63900]: pgmap v441: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:03.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:02 smithi077 ceph-mon[50325]: pgmap v441: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:03.568 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "mon": { 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": { 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": { 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "mds": { 2024-04-27T02:27:03.569 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T02:27:03.570 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:27:03.570 INFO:teuthology.orchestra.run.smithi008.stdout: "overall": { 2024-04-27T02:27:03.570 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T02:27:03.570 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:27:03.570 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-04-27T02:27:03.876 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:03 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/745319295' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:04.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-27T02:27:04.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:03 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/745319295' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:04.830 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:04 smithi008 ceph-mon[63900]: pgmap v442: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:27:05.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:04 smithi077 ceph-mon[50325]: pgmap v442: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:27:06.347 INFO:teuthology.orchestra.run.smithi008.stdout:true 2024-04-27T02:27:06.967 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:06 smithi008 ceph-mon[63900]: pgmap v443: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:06.968 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:06 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/125185603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:06.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-27T02:27:07.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:06 smithi077 ceph-mon[50325]: pgmap v443: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:07.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:06 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/125185603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:08.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:08 smithi008 ceph-mon[63900]: pgmap v444: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:09.057 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)" 2024-04-27T02:27:09.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:08 smithi077 ceph-mon[50325]: pgmap v444: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:09.666 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-04-27T02:27:09.667 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-04-27T02:27:09.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T02:27:09.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:09 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2901347820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:10.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:09 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2901347820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T02:27:10.943 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:10 smithi008 ceph-mon[63900]: pgmap v445: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:11.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:10 smithi077 ceph-mon[50325]: pgmap v445: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:11.632 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-04-27T02:27:11.632 INFO:teuthology.orchestra.run.smithi008.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T02:03:49.512940Z", "last_refresh": "2024-04-27T02:25:24.854821Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T02:03:43.278476Z", "last_refresh": "2024-04-27T02:24:32.789169Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-04-27T02:03:46.272790Z", "last_refresh": "2024-04-27T02:25:24.855012Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T02:10:19.147042Z", "last_refresh": "2024-04-27T02:24:32.789838Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T02:03:41.208567Z", "last_refresh": "2024-04-27T02:24:32.789285Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi008:172.21.15.8=smithi008", "smithi077:172.21.15.77=smithi077"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T02:04:30.227378Z", "last_refresh": "2024-04-27T02:24:32.789370Z", "running": 2, "size": 2}}, {"events": ["2024-04-27T02:26:31.166392Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-04-27T02:10:37.129302Z", "last_refresh": "2024-04-27T02:24:32.789927Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T02:03:48.259486Z", "last_refresh": "2024-04-27T02:24:32.789451Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", "last_refresh": "2024-04-27T02:24:32.789530Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T02:03:44.990122Z", "last_refresh": "2024-04-27T02:25:24.855173Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T02:27:12.252 INFO:tasks.cephadm:nfs.foo has 2/2 2024-04-27T02:27:12.253 DEBUG:teuthology.parallel:result is None 2024-04-27T02:27:12.253 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T02:27:12.262 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:27:12.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-04-27T02:27:12.290 INFO:teuthology.orchestra.run.smithi008.stderr:+ umount /mnt/foo 2024-04-27T02:27:12.302 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:27:12.313 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:27:12.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-04-27T02:27:12.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:12 smithi008 ceph-mon[63900]: pgmap v446: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:12.970 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:12 smithi008 ceph-mon[63900]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:27:13.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:12 smithi077 ceph-mon[50325]: pgmap v446: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:13.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:12 smithi077 ceph-mon[50325]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T02:27:14.363 INFO:teuthology.orchestra.run.smithi008.stdout: "foo" 2024-04-27T02:27:14.962 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:14 smithi008 ceph-mon[63900]: pgmap v447: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:27:14.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export ls foo --detailed' 2024-04-27T02:27:15.034 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:14 smithi077 ceph-mon[50325]: pgmap v447: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T02:27:15.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:15 smithi008 ceph-mon[63900]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:27:16.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:15 smithi077 ceph-mon[50325]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:27:16.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:16 smithi008 ceph-mon[63900]: pgmap v448: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:16.906 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:16 smithi008 ceph-mon[63900]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:27:17.047 INFO:teuthology.orchestra.run.smithi008.stdout:[ 2024-04-27T02:27:17.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-04-27T02:27:17.047 INFO:teuthology.orchestra.run.smithi008.stdout: "access_type": "RW", 2024-04-27T02:27:17.047 INFO:teuthology.orchestra.run.smithi008.stdout: "clients": [], 2024-04-27T02:27:17.047 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_id": "foo", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "export_id": 1, 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "fsal": { 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "fs_name": "foofs", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "name": "CEPH", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "user_id": "nfs.foo.1" 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "path": "/", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "protocols": [ 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "pseudo": "/fake", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "security_label": true, 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "squash": "none", 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "transports": [ 2024-04-27T02:27:17.048 INFO:teuthology.orchestra.run.smithi008.stdout: "TCP" 2024-04-27T02:27:17.049 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-04-27T02:27:17.049 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-04-27T02:27:17.049 INFO:teuthology.orchestra.run.smithi008.stdout:] 2024-04-27T02:27:17.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:16 smithi077 ceph-mon[50325]: pgmap v448: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:17.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:16 smithi077 ceph-mon[50325]: from='mgr.34104 172.21.15.8:0/3579201709' entity='mgr.smithi008.exkhsc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T02:27:17.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'rados -p .nfs --all ls -' 2024-04-27T02:27:19.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:18 smithi008 ceph-mon[63900]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:27:19.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:18 smithi008 ceph-mon[63900]: pgmap v449: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:19.030 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:18 smithi008 ceph-mon[63900]: mgrmap e43: smithi008.exkhsc(active, since 11m), standbys: smithi077.snksrp 2024-04-27T02:27:19.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:18 smithi077 ceph-mon[50325]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T02:27:19.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:18 smithi077 ceph-mon[50325]: pgmap v449: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:19.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:18 smithi077 ceph-mon[50325]: mgrmap e43: smithi008.exkhsc(active, since 11m), standbys: smithi077.snksrp 2024-04-27T02:27:19.358 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.1 2024-04-27T02:27:19.358 INFO:teuthology.orchestra.run.smithi008.stdout:foo grace 2024-04-27T02:27:19.358 INFO:teuthology.orchestra.run.smithi008.stdout:foo rec-0000000000000004:nfs.foo.0 2024-04-27T02:27:19.358 INFO:teuthology.orchestra.run.smithi008.stdout:foo conf-nfs.foo 2024-04-27T02:27:19.358 INFO:teuthology.orchestra.run.smithi008.stdout:foo export-1 2024-04-27T02:27:20.076 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T02:27:20.088 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:27:20.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 -- bash -c 'set -ex 2024-04-27T02:27:20.088 DEBUG:teuthology.orchestra.run.smithi008:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-04-27T02:27:20.088 DEBUG:teuthology.orchestra.run.smithi008:> ' 2024-04-27T02:27:21.077 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:20 smithi008 ceph-mon[63900]: pgmap v450: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:21.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:20 smithi077 ceph-mon[50325]: pgmap v450: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T02:27:21.881 INFO:teuthology.orchestra.run.smithi008.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-04-27T02:27:22.148 INFO:teuthology.orchestra.run.smithi008.stderr:+ [[ 6 -gt 2 ]] 2024-04-27T02:27:22.764 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T02:27:22.775 INFO:tasks.cephadm:Teardown begin 2024-04-27T02:27:22.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:27:22.803 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:27:22.835 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T02:27:22.836 DEBUG:teuthology.orchestra.run.smithi008:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T02:27:22.861 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T02:27:22.893 INFO:tasks.cephadm:Stopping all daemons... 2024-04-27T02:27:22.894 INFO:tasks.cephadm.mon.smithi008:Stopping mon.smithi008... 2024-04-27T02:27:22.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008 2024-04-27T02:27:23.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:22 smithi008 ceph-mon[63900]: pgmap v451: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T02:27:23.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:22 smithi008 ceph-mon[63900]: from='client.? 172.21.15.8:0/2341528128' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-27T02:27:23.156 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:22 smithi008 systemd[1]: Stopping Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:23.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:22 smithi077 ceph-mon[50325]: pgmap v451: 97 pgs: 97 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T02:27:23.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:22 smithi077 ceph-mon[50325]: from='client.? 172.21.15.8:0/2341528128' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-27T02:27:23.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[63896]: 2024-04-27T02:27:23.231+0000 7f1f87889700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi008 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:23.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008[63896]: 2024-04-27T02:27:23.231+0000 7f1f87889700 -1 mon.smithi008@0(leader) e3 *** Got Signal Terminated *** 2024-04-27T02:27:23.557 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 podman[108270]: 2024-04-27 02:27:23.282714588 +0000 UTC m=+0.169426925 container died 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-27T02:27:23.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 podman[108270]: 2024-04-27 02:27:23.706449787 +0000 UTC m=+0.593162111 container cleanup 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5) 2024-04-27T02:27:23.874 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 bash[108270]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008 2024-04-27T02:27:24.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service' 2024-04-27T02:27:24.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:23 smithi008 podman[108296]: 2024-04-27 02:27:23.899851014 +0000 UTC m=+0.619730424 container remove 50e74d82ab4df0ecf68b0894beeeefd32821ded538f705f5cb01a8abfc2f1076 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi008, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T02:27:24.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:24 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Deactivated successfully. 2024-04-27T02:27:24.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:24 smithi008 systemd[1]: Stopped Ceph mon.smithi008 for 27e8cf32-043a-11ef-bc93-c7b262605968. 2024-04-27T02:27:24.128 INFO:journalctl@ceph.mon.smithi008.smithi008.stdout:Apr 27 02:27:24 smithi008 systemd[1]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi008.service: Consumed 15.310s CPU time. 2024-04-27T02:27:24.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:24.159 INFO:tasks.cephadm.mon.smithi008:Stopped mon.smithi008 2024-04-27T02:27:24.159 INFO:tasks.cephadm.mon.smithi077:Stopping mon.smithi077... 2024-04-27T02:27:24.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi077 2024-04-27T02:27:24.444 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:24.731 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077[50321]: 2024-04-27T02:27:24.468+0000 7fcbd7a7a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:24.731 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077[50321]: 2024-04-27T02:27:24.468+0000 7fcbd7a7a700 -1 mon.smithi077@1(peon) e3 *** Got Signal Terminated *** 2024-04-27T02:27:24.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 podman[75784]: 2024-04-27 02:27:24.48980875 +0000 UTC m=+0.115155280 container died 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-27T02:27:25.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 podman[75784]: 2024-04-27 02:27:24.94474455 +0000 UTC m=+0.570091120 container cleanup 288f6e8c6e2e84c170ef70366c3941cf0c5924e1a9d079a6ad570d34c55302b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:27:25.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Apr 27 02:27:24 smithi077 bash[75784]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-mon-smithi077 2024-04-27T02:27:25.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@mon.smithi077.service' 2024-04-27T02:27:25.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:25.383 INFO:tasks.cephadm.mon.smithi077:Stopped mon.smithi077 2024-04-27T02:27:25.383 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-27T02:27:25.383 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.0 2024-04-27T02:27:25.694 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:25 smithi008 systemd[1]: Stopping Ceph osd.0 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:26.156 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:25 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:27:25.719+0000 7ff7e92e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:26.156 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:25 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:27:25.719+0000 7ff7e92e6700 -1 osd.0 156 *** Got signal Terminated *** 2024-04-27T02:27:26.156 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:25 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0[69981]: 2024-04-27T02:27:25.719+0000 7ff7e92e6700 -1 osd.0 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:27:31.504 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:31 smithi008 podman[108402]: 2024-04-27 02:27:31.244264982 +0000 UTC m=+5.639229643 container died f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T02:27:31.871 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:31 smithi008 podman[108402]: 2024-04-27 02:27:31.620003515 +0000 UTC m=+6.014968162 container cleanup f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T02:27:31.871 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:31 smithi008 bash[108402]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0 2024-04-27T02:27:32.156 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:31 smithi008 podman[108420]: 2024-04-27 02:27:31.896474222 +0000 UTC m=+0.642922522 container remove f1128c0b1706d6efa34753238cfef952c3f8e7790d847e05a831ddc580e77b2c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-27T02:27:32.477 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.153801803 +0000 UTC m=+0.019014128 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:27:32.477 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.27254823 +0000 UTC m=+0.137760577 container create 9fa8330405d3916b05b66bf1269c037b83469948bde0c37d5fdad270e03d98db (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-27T02:27:32.477 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.502408128 +0000 UTC m=+0.367620433 container init 9fa8330405d3916b05b66bf1269c037b83469948bde0c37d5fdad270e03d98db (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T02:27:32.788 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.505932999 +0000 UTC m=+0.371145314 container start 9fa8330405d3916b05b66bf1269c037b83469948bde0c37d5fdad270e03d98db (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:27:32.788 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.610599363 +0000 UTC m=+0.475811679 container attach 9fa8330405d3916b05b66bf1269c037b83469948bde0c37d5fdad270e03d98db (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:27:33.106 INFO:journalctl@ceph.osd.0.smithi008.stdout:Apr 27 02:27:32 smithi008 podman[108511]: 2024-04-27 02:27:32.814301155 +0000 UTC m=+0.679513461 container died 9fa8330405d3916b05b66bf1269c037b83469948bde0c37d5fdad270e03d98db (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:27:33.429 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.0.service' 2024-04-27T02:27:33.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:33.468 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-27T02:27:33.469 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-27T02:27:33.469 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.1 2024-04-27T02:27:33.785 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:33 smithi008 systemd[1]: Stopping Ceph osd.1 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:34.156 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:33 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:27:33.809+0000 7f37b4224700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:34.156 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:33 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:27:33.809+0000 7f37b4224700 -1 osd.1 156 *** Got signal Terminated *** 2024-04-27T02:27:34.156 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:33 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1[72678]: 2024-04-27T02:27:33.809+0000 7f37b4224700 -1 osd.1 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:27:39.653 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:39 smithi008 podman[108617]: 2024-04-27 02:27:39.394534397 +0000 UTC m=+5.672831264 container died 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T02:27:39.906 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:39 smithi008 podman[108617]: 2024-04-27 02:27:39.752067683 +0000 UTC m=+6.030364548 container cleanup 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:27:39.906 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:39 smithi008 bash[108617]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1 2024-04-27T02:27:40.360 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108666]: 2024-04-27 02:27:40.020178648 +0000 UTC m=+0.624412567 container remove 623e09212f515d066862bb5b62851555f6f05e7400b344f2a60ae819ad660445 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:27:40.656 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.289126607 +0000 UTC m=+0.018955964 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:27:40.656 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.387696993 +0000 UTC m=+0.117526340 container create 7b964a6b43011ee5f8a9a8a5ca8b8b23d42a718cc7c00e4476b3fbb198597918 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T02:27:40.656 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.598426789 +0000 UTC m=+0.328256139 container init 7b964a6b43011ee5f8a9a8a5ca8b8b23d42a718cc7c00e4476b3fbb198597918 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T02:27:40.656 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.601709529 +0000 UTC m=+0.331538881 container start 7b964a6b43011ee5f8a9a8a5ca8b8b23d42a718cc7c00e4476b3fbb198597918 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:27:41.156 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.698593686 +0000 UTC m=+0.428423034 container attach 7b964a6b43011ee5f8a9a8a5ca8b8b23d42a718cc7c00e4476b3fbb198597918 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-27T02:27:41.156 INFO:journalctl@ceph.osd.1.smithi008.stdout:Apr 27 02:27:40 smithi008 podman[108759]: 2024-04-27 02:27:40.904721835 +0000 UTC m=+0.634551190 container died 7b964a6b43011ee5f8a9a8a5ca8b8b23d42a718cc7c00e4476b3fbb198597918 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T02:27:41.555 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.1.service' 2024-04-27T02:27:41.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:41.597 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-27T02:27:41.597 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-27T02:27:41.597 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.2 2024-04-27T02:27:41.906 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:41 smithi008 systemd[1]: Stopping Ceph osd.2 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:42.406 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:41 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:27:41.967+0000 7fb67ae0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:42.406 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:41 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:27:41.967+0000 7fb67ae0d700 -1 osd.2 156 *** Got signal Terminated *** 2024-04-27T02:27:42.406 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:41 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2[75252]: 2024-04-27T02:27:41.967+0000 7fb67ae0d700 -1 osd.2 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:27:47.875 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:47 smithi008 podman[108863]: 2024-04-27 02:27:47.565962756 +0000 UTC m=+5.709637965 container died 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:27:48.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:47 smithi008 podman[108863]: 2024-04-27 02:27:47.989540725 +0000 UTC m=+6.133215925 container cleanup 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T02:27:48.133 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:47 smithi008 bash[108863]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2 2024-04-27T02:27:48.406 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108880]: 2024-04-27 02:27:48.157816985 +0000 UTC m=+0.593264463 container remove 35f27121d1988b6ee576e06a7530c2be35dd3221bab2b5bd36938144d6797419 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-27T02:27:48.894 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108975]: 2024-04-27 02:27:48.517452779 +0000 UTC m=+0.018954159 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:27:48.894 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108975]: 2024-04-27 02:27:48.633539314 +0000 UTC m=+0.135040713 container create ea08276c61adef64ce08f09ce89157ac72bc1a94dcc8ed0c9c290d43fd453b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T02:27:48.894 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108975]: 2024-04-27 02:27:48.84473745 +0000 UTC m=+0.346238809 container init ea08276c61adef64ce08f09ce89157ac72bc1a94dcc8ed0c9c290d43fd453b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-27T02:27:48.894 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108975]: 2024-04-27 02:27:48.848474708 +0000 UTC m=+0.349976069 container start ea08276c61adef64ce08f09ce89157ac72bc1a94dcc8ed0c9c290d43fd453b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb) 2024-04-27T02:27:48.894 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:48 smithi008 podman[108975]: 2024-04-27 02:27:48.919289619 +0000 UTC m=+0.420790982 container attach ea08276c61adef64ce08f09ce89157ac72bc1a94dcc8ed0c9c290d43fd453b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-27T02:27:49.156 INFO:journalctl@ceph.osd.2.smithi008.stdout:Apr 27 02:27:49 smithi008 podman[108975]: 2024-04-27 02:27:49.161083949 +0000 UTC m=+0.662585315 container died ea08276c61adef64ce08f09ce89157ac72bc1a94dcc8ed0c9c290d43fd453b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T02:27:49.756 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.2.service' 2024-04-27T02:27:49.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:49.799 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-27T02:27:49.799 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-27T02:27:49.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.3 2024-04-27T02:27:50.126 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:49 smithi008 systemd[1]: Stopping Ceph osd.3 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:50.406 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:50 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:50.150+0000 7f4350b71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:50.406 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:50 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:50.150+0000 7f4350b71700 -1 osd.3 156 *** Got signal Terminated *** 2024-04-27T02:27:50.406 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:50 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:50.150+0000 7f4350b71700 -1 osd.3 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:27:52.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:52 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:52.142+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:52.906 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:52 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:52.666+0000 7f434915d700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:29.636780+0000 front 2024-04-27T02:27:29.636910+0000 (oldest deadline 2024-04-27T02:27:51.936843+0000) 2024-04-27T02:27:53.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:52 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:52.859+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:53.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:53.101+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:53.877 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:53.744+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:54.156 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:53 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:53.691+0000 7f434915d700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:29.636780+0000 front 2024-04-27T02:27:29.636910+0000 (oldest deadline 2024-04-27T02:27:51.936843+0000) 2024-04-27T02:27:54.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:53 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:53.901+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:54.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:54 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:54.111+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:54.625 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:54 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:54.298+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:27:54.881 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:54 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:54.793+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:55.156 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:54 smithi008 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3[77908]: 2024-04-27T02:27:54.687+0000 7f434915d700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:29.636780+0000 front 2024-04-27T02:27:29.636910+0000 (oldest deadline 2024-04-27T02:27:51.936843+0000) 2024-04-27T02:27:55.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:54 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:54.906+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:55.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:55 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:55.155+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:55.731 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:55 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:55.291+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:27:55.993 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:55 smithi008 podman[109080]: 2024-04-27 02:27:55.765765682 +0000 UTC m=+5.713152873 container died 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:27:56.082 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:55 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:55.896+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:56.082 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:55 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:55.797+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:56.259 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109080]: 2024-04-27 02:27:56.092208778 +0000 UTC m=+6.039595976 container cleanup 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T02:27:56.260 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 bash[109080]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3 2024-04-27T02:27:56.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:56 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:56.335+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:27:56.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:56 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:56.107+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:56.656 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109099]: 2024-04-27 02:27:56.285024031 +0000 UTC m=+0.516183760 container remove 00420044291b07711c5b335541fa76965cc0f30e89c6564e34df8ee7b9216d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T02:27:56.987 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109223]: 2024-04-27 02:27:56.637750605 +0000 UTC m=+0.018648021 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:27:56.988 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109223]: 2024-04-27 02:27:56.744174362 +0000 UTC m=+0.125071770 container create b4b7ca15164feb4bd820fa8fd317141a93d1d1df2cc6205d1c13012185ff02d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T02:27:56.988 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109223]: 2024-04-27 02:27:56.938680839 +0000 UTC m=+0.319578248 container init b4b7ca15164feb4bd820fa8fd317141a93d1d1df2cc6205d1c13012185ff02d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:27:56.988 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:56 smithi008 podman[109223]: 2024-04-27 02:27:56.942599245 +0000 UTC m=+0.323496648 container start b4b7ca15164feb4bd820fa8fd317141a93d1d1df2cc6205d1c13012185ff02d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T02:27:56.988 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:57 smithi008 podman[109223]: 2024-04-27 02:27:57.013168273 +0000 UTC m=+0.394065676 container attach b4b7ca15164feb4bd820fa8fd317141a93d1d1df2cc6205d1c13012185ff02d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-27T02:27:57.040 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:56 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:56.896+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:57.041 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:56 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:56.767+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:57.406 INFO:journalctl@ceph.osd.3.smithi008.stdout:Apr 27 02:27:57 smithi008 podman[109223]: 2024-04-27 02:27:57.240487115 +0000 UTC m=+0.621384518 container died b4b7ca15164feb4bd820fa8fd317141a93d1d1df2cc6205d1c13012185ff02d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-27T02:27:57.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:57 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:57.286+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:27:57.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:57 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:57.065+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:57.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.3.service' 2024-04-27T02:27:57.844 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:27:57.844 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-27T02:27:57.844 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-27T02:27:57.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.4 2024-04-27T02:27:58.088 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:57 smithi077 systemd[1]: Stopping Ceph osd.4 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:27:58.088 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:57 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:57.925+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:58.089 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:57 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:57.778+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:58.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:58.112+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:58.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:58.164+0000 7f9c06892700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:27:58.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:58.164+0000 7f9c06892700 -1 osd.4 156 *** Got signal Terminated *** 2024-04-27T02:27:58.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:58.164+0000 7f9c06892700 -1 osd.4 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:27:58.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:58.283+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:27:59.090 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:58.883+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:27:59.090 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:58 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:58.770+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:27:59.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:27:59 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:27:59.114+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:27:59.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:27:59 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:27:59.292+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:28:00.133 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:27:59 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:27:59.820+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:00.134 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:27:59 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:27:59.894+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:00.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:28:00.336+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:28:00.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:00.158+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:01.148 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:00.786+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:01.148 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:00.877+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:01.148 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:00 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:00.877+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:01.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:01.172+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:01.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:28:01.344+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:28:01.482 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:28:01.344+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.558867+0000 front 2024-04-27T02:27:36.558996+0000 (oldest deadline 2024-04-27T02:28:01.258626+0000) 2024-04-27T02:28:01.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:01.737+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:01.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:01.849+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:01.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:01 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:01.849+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:02.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:02.143+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:02.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:28:02.323+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:28.357845+0000 front 2024-04-27T02:27:28.357778+0000 (oldest deadline 2024-04-27T02:27:54.257694+0000) 2024-04-27T02:28:02.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4[55267]: 2024-04-27T02:28:02.323+0000 7f9bfee7e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.558867+0000 front 2024-04-27T02:27:36.558996+0000 (oldest deadline 2024-04-27T02:28:01.258626+0000) 2024-04-27T02:28:02.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:02.847+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:02.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:02.847+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:02.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:02.755+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:02.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:02 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:02.755+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:03.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:03.172+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:03.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:03.172+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:03.864 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:03 smithi077 podman[75983]: 2024-04-27 02:28:03.616375986 +0000 UTC m=+5.555941794 container died 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T02:28:03.865 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:03.760+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:03.865 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:03.760+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:04.132 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[75983]: 2024-04-27 02:28:04.073645837 +0000 UTC m=+6.013211637 container cleanup 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:28:04.132 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 bash[75983]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4 2024-04-27T02:28:04.132 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:03.889+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:04.132 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:03 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:03.889+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:04.481 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[76000]: 2024-04-27 02:28:04.341864408 +0000 UTC m=+0.720211698 container remove 5b45fe64c65bbb16907712515ff541f351c5e49eddf8a7f5e1d76ee39733dbd2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-27T02:28:04.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:04.172+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:04.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:04.172+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[76089]: 2024-04-27 02:28:04.593843986 +0000 UTC m=+0.017062183 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[76089]: 2024-04-27 02:28:04.718063426 +0000 UTC m=+0.141281668 container create 5d7a518afeb781c37d3ecb0b7c7e1edba3d87f7b4cd69f913aea90e1da1cb345 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[76089]: 2024-04-27 02:28:04.929292264 +0000 UTC m=+0.352510461 container init 5d7a518afeb781c37d3ecb0b7c7e1edba3d87f7b4cd69f913aea90e1da1cb345 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:04 smithi077 podman[76089]: 2024-04-27 02:28:04.932384836 +0000 UTC m=+0.355603036 container start 5d7a518afeb781c37d3ecb0b7c7e1edba3d87f7b4cd69f913aea90e1da1cb345 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:04.937+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:04.982 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:04.937+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:04.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:04.739+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:04.983 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:04 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:04.739+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:05.394 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:05.151+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:05.394 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:05.151+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:05.395 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:05 smithi077 podman[76089]: 2024-04-27 02:28:05.029359063 +0000 UTC m=+0.452577276 container attach 5d7a518afeb781c37d3ecb0b7c7e1edba3d87f7b4cd69f913aea90e1da1cb345 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-27T02:28:05.395 INFO:journalctl@ceph.osd.4.smithi077.stdout:Apr 27 02:28:05 smithi077 podman[76089]: 2024-04-27 02:28:05.240443197 +0000 UTC m=+0.663661399 container died 5d7a518afeb781c37d3ecb0b7c7e1edba3d87f7b4cd69f913aea90e1da1cb345 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-27T02:28:05.916 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.4.service' 2024-04-27T02:28:05.953 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:28:05.953 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-27T02:28:05.954 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-27T02:28:05.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.5 2024-04-27T02:28:06.014 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:05.890+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:06.014 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:05.890+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:06.014 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:05.755+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:06.014 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:05 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:05.755+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:06.295 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 systemd[1]: Stopping Ceph osd.5 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:28:06.296 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:06.143+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:06.296 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:06.143+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:06.712 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:06.320+0000 7f87e7126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:28:06.712 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:06.320+0000 7f87e7126700 -1 osd.5 156 *** Got signal Terminated *** 2024-04-27T02:28:06.713 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:06.320+0000 7f87e7126700 -1 osd.5 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:28:06.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:06.871+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:06.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:06.871+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:06.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:06.737+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:06.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:06 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:06.737+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:07.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:07.175+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:07.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:07.175+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:07.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:07.892+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:07.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:07.892+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:07.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:07.730+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:07.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:07 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:07.730+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:08.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:08.142+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:08.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:08.142+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:08.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:08.917+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:08.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:08.917+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:08.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:08.706+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:08.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:08 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:08.706+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:09.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:09.159+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:09.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:09.159+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:09.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:09.159+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:09.928 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:09.678+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:09.928 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:09.678+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:10.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:09.953+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:10.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:09 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:09.953+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:10.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:10.174+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:10.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:10.174+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:10.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:10.174+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:10.971 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:10.654+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:10.971 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:10.654+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:10.971 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:10.654+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:11.231 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:10.995+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:30.442090+0000 front 2024-04-27T02:27:30.442040+0000 (oldest deadline 2024-04-27T02:27:52.141870+0000) 2024-04-27T02:28:11.232 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:10.995+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.043088+0000 front 2024-04-27T02:27:38.043137+0000 (oldest deadline 2024-04-27T02:28:00.342924+0000) 2024-04-27T02:28:11.232 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:10 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5[57547]: 2024-04-27T02:28:10.995+0000 7f87e371a700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.944264+0000 front 2024-04-27T02:27:44.944359+0000 (oldest deadline 2024-04-27T02:28:10.844306+0000) 2024-04-27T02:28:11.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:11.126+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:11.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:11.126+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:11.232 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:11.126+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:11.981 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:11 smithi077 podman[76205]: 2024-04-27 02:28:11.870751018 +0000 UTC m=+5.664690310 container died a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:28:11.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:11.647+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:11.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:11.647+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:11.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:11 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:11.647+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:12.480 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:12 smithi077 podman[76205]: 2024-04-27 02:28:12.296339118 +0000 UTC m=+6.090278377 container cleanup a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:28:12.480 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:12 smithi077 bash[76205]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5 2024-04-27T02:28:12.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:12.108+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:12.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:12.108+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:12.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:12.108+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:12.731 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:12 smithi077 podman[76223]: 2024-04-27 02:28:12.506025323 +0000 UTC m=+0.628810183 container remove a8ae0b8667e3c817e5e6677bc8a5ceafca47e6bea7fa7414666b86497c363f90 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T02:28:12.731 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:12.599+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:12.731 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:12.599+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:12.732 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:12 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:12.599+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:13.145 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:13.153+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:13.145 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:13.153+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:13.146 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:13.153+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:13.146 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:12 smithi077 podman[76314]: 2024-04-27 02:28:12.823557295 +0000 UTC m=+0.018120881 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:28:13.146 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:12 smithi077 podman[76314]: 2024-04-27 02:28:12.932091964 +0000 UTC m=+0.126655539 container create 14196e7887a217af4b5d1f3b8d13f86a1a2b1b7a0b4258e9501fdc84e75d33cc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:28:13.146 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:13 smithi077 podman[76314]: 2024-04-27 02:28:13.16816253 +0000 UTC m=+0.362726107 container init 14196e7887a217af4b5d1f3b8d13f86a1a2b1b7a0b4258e9501fdc84e75d33cc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-27T02:28:13.436 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:13 smithi077 podman[76314]: 2024-04-27 02:28:13.171219723 +0000 UTC m=+0.365783299 container start 14196e7887a217af4b5d1f3b8d13f86a1a2b1b7a0b4258e9501fdc84e75d33cc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T02:28:13.437 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:13 smithi077 podman[76314]: 2024-04-27 02:28:13.259889889 +0000 UTC m=+0.454453491 container attach 14196e7887a217af4b5d1f3b8d13f86a1a2b1b7a0b4258e9501fdc84e75d33cc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T02:28:13.721 INFO:journalctl@ceph.osd.5.smithi077.stdout:Apr 27 02:28:13 smithi077 podman[76314]: 2024-04-27 02:28:13.462461396 +0000 UTC m=+0.657024963 container died 14196e7887a217af4b5d1f3b8d13f86a1a2b1b7a0b4258e9501fdc84e75d33cc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:28:13.721 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:13.577+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:13.721 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:13.577+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:13.721 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:13 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:13.577+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:14.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.5.service' 2024-04-27T02:28:14.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:28:14.136 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-27T02:28:14.136 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-27T02:28:14.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.6 2024-04-27T02:28:14.458 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 systemd[1]: Stopping Ceph osd.6 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.483+0000 7fca8ec3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.483+0000 7fca8ec3d700 -1 osd.6 156 *** Got signal Terminated *** 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.483+0000 7fca8ec3d700 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:14.158+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:14.158+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:14.459 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:14.158+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:14.731 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.582+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:14.732 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.582+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:14.732 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:14 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:14.582+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:15.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:15.125+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:15.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:15.125+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:15.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:15.125+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:15.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:15.538+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:15.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:15.538+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:15.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:15 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:15.538+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:16.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:16.099+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:16.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:16.099+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:16.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:16.099+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:16.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:16.564+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:16.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:16.564+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:16.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:16 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:16.564+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:17.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:17.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:17.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:17.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:17.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:17.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:17.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:17.525+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:17.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:17.525+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:17.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:17 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:17.525+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:18.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:18.066+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:18.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:18.066+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:18.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:18.066+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:18.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:18.516+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:26.899370+0000 front 2024-04-27T02:27:26.899178+0000 (oldest deadline 2024-04-27T02:27:52.799218+0000) 2024-04-27T02:28:18.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:18.516+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:36.899997+0000 front 2024-04-27T02:27:36.900154+0000 (oldest deadline 2024-04-27T02:28:02.700358+0000) 2024-04-27T02:28:18.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:18.516+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:44.400953+0000 front 2024-04-27T02:27:44.401119+0000 (oldest deadline 2024-04-27T02:28:10.301013+0000) 2024-04-27T02:28:18.982 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:18 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6[59807]: 2024-04-27T02:28:18.516+0000 7fca8b231700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:55.001853+0000 front 2024-04-27T02:27:55.001885+0000 (oldest deadline 2024-04-27T02:28:18.501813+0000) 2024-04-27T02:28:19.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:19 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:19.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:19.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:19 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:19.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:19.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:19 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:19.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:20.286 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:20 smithi077 podman[76420]: 2024-04-27 02:28:20.040376142 +0000 UTC m=+5.657593139 container died d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, RELEASE=squid-b22e2eb) 2024-04-27T02:28:20.286 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:20.127+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:20.286 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:20.127+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:20.286 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:20.127+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:20.286 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:20 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:20.127+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:20.659 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:20 smithi077 podman[76420]: 2024-04-27 02:28:20.408447238 +0000 UTC m=+6.025664260 container cleanup d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-27T02:28:20.659 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:20 smithi077 bash[76420]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6 2024-04-27T02:28:20.981 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:20 smithi077 podman[76470]: 2024-04-27 02:28:20.684724159 +0000 UTC m=+0.637622912 container remove d80d5951ed26101acb8c4215fbe90aebfc1fceacbedeee42b09c28a514b8b7a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T02:28:21.246 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:21 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:21.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:21 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:21.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:21 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:21.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:21 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:21.115+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:20.915805841 +0000 UTC m=+0.017647193 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:21.029476017 +0000 UTC m=+0.131317352 container create 15a4fb17ac737636ebf6ba7ba410dd1818a472ad981d8b488b3882d23488864b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-27T02:28:21.247 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:21.272157825 +0000 UTC m=+0.373999164 container init 15a4fb17ac737636ebf6ba7ba410dd1818a472ad981d8b488b3882d23488864b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T02:28:21.544 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:21.275302102 +0000 UTC m=+0.377143440 container start 15a4fb17ac737636ebf6ba7ba410dd1818a472ad981d8b488b3882d23488864b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-27T02:28:21.544 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:21.355532385 +0000 UTC m=+0.457373737 container attach 15a4fb17ac737636ebf6ba7ba410dd1818a472ad981d8b488b3882d23488864b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T02:28:21.833 INFO:journalctl@ceph.osd.6.smithi077.stdout:Apr 27 02:28:21 smithi077 podman[76563]: 2024-04-27 02:28:21.569910666 +0000 UTC m=+0.671752011 container died 15a4fb17ac737636ebf6ba7ba410dd1818a472ad981d8b488b3882d23488864b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-27T02:28:22.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.6.service' 2024-04-27T02:28:22.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:28:22.158 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-27T02:28:22.158 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-27T02:28:22.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.7 2024-04-27T02:28:22.470 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.113+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:22.470 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.113+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:22.470 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.113+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:22.471 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.113+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:22.471 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 systemd[1]: Stopping Ceph osd.7 for 27e8cf32-043a-11ef-bc93-c7b262605968... 2024-04-27T02:28:22.471 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.494+0000 7f9ff608c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T02:28:22.471 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.494+0000 7f9ff608c700 -1 osd.7 156 *** Got signal Terminated *** 2024-04-27T02:28:22.471 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:22 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:22.494+0000 7f9ff608c700 -1 osd.7 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T02:28:23.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:23 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:23.093+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:23.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:23 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:23.093+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:23.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:23 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:23.093+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:23.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:23 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:23.093+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:24.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:24.052+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:24.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:24.052+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:24.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:24.052+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:24.231 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:24 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:24.052+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:25.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:25 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:25.062+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:25.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:25 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:25.062+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:25.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:25 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:25.062+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:25.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:25 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:25.062+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:26.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:26 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:26.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:26.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:26 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:26.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:26.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:26 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:26.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:26.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:26 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:26.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:26.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:26 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:26.098+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.77:6804 osd.4 since back 2024-04-27T02:28:03.143786+0000 front 2024-04-27T02:28:03.143802+0000 (oldest deadline 2024-04-27T02:28:25.443706+0000) 2024-04-27T02:28:27.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:27 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:27.090+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-04-27T02:27:27.340407+0000 front 2024-04-27T02:27:27.340643+0000 (oldest deadline 2024-04-27T02:27:52.040428+0000) 2024-04-27T02:28:27.481 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:27 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:27.090+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-04-27T02:27:38.441494+0000 front 2024-04-27T02:27:38.441567+0000 (oldest deadline 2024-04-27T02:28:03.141482+0000) 2024-04-27T02:28:27.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:27 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:27.090+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-04-27T02:27:43.142155+0000 front 2024-04-27T02:27:43.142102+0000 (oldest deadline 2024-04-27T02:28:09.042126+0000) 2024-04-27T02:28:27.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:27 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:27.090+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-04-27T02:27:54.342874+0000 front 2024-04-27T02:27:54.342857+0000 (oldest deadline 2024-04-27T02:28:19.642854+0000) 2024-04-27T02:28:27.482 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:27 smithi077 ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7[62066]: 2024-04-27T02:28:27.090+0000 7f9fee678700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.77:6804 osd.4 since back 2024-04-27T02:28:03.143786+0000 front 2024-04-27T02:28:03.143802+0000 (oldest deadline 2024-04-27T02:28:25.443706+0000) 2024-04-27T02:28:28.345 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:28 smithi077 podman[76668]: 2024-04-27 02:28:28.10820251 +0000 UTC m=+5.708949918 container died 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb) 2024-04-27T02:28:28.712 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:28 smithi077 podman[76668]: 2024-04-27 02:28:28.478755473 +0000 UTC m=+6.079502879 container cleanup 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-27T02:28:28.713 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:28 smithi077 bash[76668]: ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7 2024-04-27T02:28:28.981 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:28 smithi077 podman[76697]: 2024-04-27 02:28:28.738140889 +0000 UTC m=+0.626133968 container remove 0baf0f4a3a036baf314fd5beea85e82f7156c72aaffdd4cd575d914b318a3fa4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-27T02:28:29.332 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:28.956159194 +0000 UTC m=+0.017719463 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T02:28:29.333 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:29.055794841 +0000 UTC m=+0.117355138 container create 6823caf58791bf53f569e9a9f9dea7f5473f6606b1b91dd92c36e3406084e55d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T02:28:29.333 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:29.283271813 +0000 UTC m=+0.344832068 container init 6823caf58791bf53f569e9a9f9dea7f5473f6606b1b91dd92c36e3406084e55d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-27T02:28:29.333 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:29.286469926 +0000 UTC m=+0.348030190 container start 6823caf58791bf53f569e9a9f9dea7f5473f6606b1b91dd92c36e3406084e55d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T02:28:29.333 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:29.35817486 +0000 UTC m=+0.419735125 container attach 6823caf58791bf53f569e9a9f9dea7f5473f6606b1b91dd92c36e3406084e55d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T02:28:29.731 INFO:journalctl@ceph.osd.7.smithi077.stdout:Apr 27 02:28:29 smithi077 podman[76789]: 2024-04-27 02:28:29.570109606 +0000 UTC m=+0.631669862 container died 6823caf58791bf53f569e9a9f9dea7f5473f6606b1b91dd92c36e3406084e55d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-27e8cf32-043a-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-27T02:28:30.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27e8cf32-043a-11ef-bc93-c7b262605968@osd.7.service' 2024-04-27T02:28:30.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T02:28:30.164 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-27T02:28:30.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T02:31:29.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T02:31:50.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:31:50.197 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T02:31:50.223 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T02:31:50.224 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi008/crash 2024-04-27T02:31:50.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash -- . 2024-04-27T02:31:50.276 INFO:teuthology.orchestra.run.smithi008.stderr:tar: /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T02:31:50.276 INFO:teuthology.orchestra.run.smithi008.stderr:tar: Error is not recoverable: exiting now 2024-04-27T02:31:50.278 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi077/crash 2024-04-27T02:31:50.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash -- . 2024-04-27T02:31:50.318 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T02:31:50.318 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-04-27T02:31:50.320 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T02:31:50.320 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T02:31:50.350 INFO:teuthology.orchestra.run.smithi008.stdout:2024-04-27T02:14:44.978233+0000 mon.smithi008 (mon.0) 882 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:31:50.351 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-27T02:31:50.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[SEC\]' /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T02:31:50.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]' /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T02:31:50.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[WRN\]' /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T02:31:50.553 INFO:teuthology.orchestra.run.smithi008.stdout:2024-04-27T02:14:44.978233+0000 mon.smithi008 (mon.0) 882 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T02:31:50.553 INFO:tasks.cephadm:Compressing logs... 2024-04-27T02:31:50.554 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T02:31:50.595 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi008.wzyiku.log 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:: No such file or directory 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi008.wzyiku.log: /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-27T02:31:50.618 INFO:teuthology.orchestra.run.smithi008.stderr: 89.4% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi008.wzyiku.log.gz 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi008.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi008.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi008.exkhsc.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr: 87.1% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T02:31:50.624 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi008.exkhsc.log: 91.5% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi077.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi077.log: 85.7% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T02:31:50.627 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T02:31:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi077.qcnfuu.log 2024-04-27T02:31:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi077.snksrp.log 2024-04-27T02:31:50.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi077.qcnfuu.log: -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T02:31:50.628 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi077.qcnfuu.log.gz 2024-04-27T02:31:50.634 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T02:31:50.640 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi077.snksrp.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T02:31:50.646 INFO:teuthology.orchestra.run.smithi077.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T02:31:50.652 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T02:31:50.666 INFO:teuthology.orchestra.run.smithi008.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T02:31:50.697 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi077.snksrp.log.gz 2024-04-27T02:31:51.034 INFO:teuthology.orchestra.run.smithi008.stderr: 88.4% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mgr.smithi008.exkhsc.log.gz 2024-04-27T02:31:51.103 INFO:teuthology.orchestra.run.smithi077.stderr: 92.6% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi077.log.gz 2024-04-27T02:31:52.101 INFO:teuthology.orchestra.run.smithi008.stderr: 91.8% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-mon.smithi008.log.gz 2024-04-27T02:32:03.403 INFO:teuthology.orchestra.run.smithi008.stderr: 94.5% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T02:32:04.876 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-27T02:32:06.694 INFO:teuthology.orchestra.run.smithi077.stderr: 94.2% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-27T02:32:08.514 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T02:32:09.924 INFO:teuthology.orchestra.run.smithi077.stderr: 94.2% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-27T02:32:10.379 INFO:teuthology.orchestra.run.smithi008.stderr: 94.2% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T02:32:10.532 INFO:teuthology.orchestra.run.smithi008.stderr: 94.1% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-27T02:32:10.534 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-04-27T02:32:10.534 INFO:teuthology.orchestra.run.smithi008.stderr:real 0m19.926s 2024-04-27T02:32:10.534 INFO:teuthology.orchestra.run.smithi008.stderr:user 1m9.616s 2024-04-27T02:32:10.534 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m2.581s 2024-04-27T02:32:15.762 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/27e8cf32-043a-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-27T02:32:15.764 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-27T02:32:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m25.154s 2024-04-27T02:32:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:user 1m12.479s 2024-04-27T02:32:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m2.762s 2024-04-27T02:32:15.764 INFO:tasks.cephadm:Archiving logs... 2024-04-27T02:32:15.765 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi008/log 2024-04-27T02:32:15.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T02:32:24.825 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi077/log 2024-04-27T02:32:24.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T02:32:35.050 INFO:tasks.cephadm:Removing cluster... 2024-04-27T02:32:35.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 --force 2024-04-27T02:32:35.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27e8cf32-043a-11ef-bc93-c7b262605968 --force 2024-04-27T02:32:35.813 INFO:tasks.cephadm:Removing cephadm ... 2024-04-27T02:32:35.813 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T02:32:35.829 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T02:32:35.843 INFO:tasks.cephadm:Teardown complete 2024-04-27T02:32:35.844 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T02:32:35.854 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T02:32:35.855 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T02:32:35.871 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T02:32:35.884 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-04-27T02:32:35.889 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T02:32:35.890 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-04-27T02:32:35.890 INFO:teuthology.orchestra.run.smithi008.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 +2435us[+2454us] +/- 53ms 2024-04-27T02:32:35.890 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -2810us[-2810us] +/- 53ms 2024-04-27T02:32:35.890 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +529us[ +529us] +/- 77ms 2024-04-27T02:32:35.890 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:32:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +2436us[+2434us] +/- 53ms 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -2828us[-2829us] +/- 53ms 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +516us[ +515us] +/- 77ms 2024-04-27T02:32:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T02:32:35.902 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T02:32:35.942 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T02:32:35.943 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T02:32:35.993 DEBUG:teuthology.orchestra.run.smithi008:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T02:32:36.072 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T02:32:36.142 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T02:32:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:32:36.171 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T02:32:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:32:36.197 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T02:32:36.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T02:32:36.304 INFO:teuthology.task.internal:Duration was 2043.771293 seconds 2024-04-27T02:32:36.305 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T02:32:36.330 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T02:32:36.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T02:32:36.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T02:32:36.368 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T02:32:36.368 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T02:32:36.766 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T02:32:36.766 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-04-27T02:32:36.766 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T02:32:36.787 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T02:32:36.788 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T02:32:36.827 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T02:32:36.827 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T02:32:36.830 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T02:32:36.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T02:32:36.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T02:32:36.897 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T02:32:51.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T02:32:51.460 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T02:32:51.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T02:32:51.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T02:32:51.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T02:32:51.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T02:32:51.550 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T02:32:51.570 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-04-27T02:32:51.579 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-04-27T02:32:51.611 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T02:32:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:32:51.646 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T02:32:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T02:32:51.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T02:32:51.677 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T02:32:51.678 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi008 2024-04-27T02:32:51.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T02:32:51.753 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541/remote/smithi077 2024-04-27T02:32:51.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T02:32:51.795 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T02:32:51.795 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T02:32:51.797 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T02:32:51.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T02:32:51.856 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T02:32:51.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T02:32:51.901 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T02:32:51.901 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T02:32:51.903 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T02:32:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 02:32 /home/ubuntu/cephtest 2024-04-27T02:32:51.919 INFO:teuthology.orchestra.run.smithi077.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 02:32 /home/ubuntu/cephtest 2024-04-27T02:32:51.920 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T02:32:51.936 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T02:32:52.145 INFO:teuthology.nuke:Checking targets against current locks 2024-04-27T02:32:52.173 DEBUG:teuthology.nuke:shortname: smithi008 2024-04-27T02:32:52.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T02:32:52.196 DEBUG:teuthology.nuke:shortname: smithi077 2024-04-27T02:32:52.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T02:32:52.199 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:52:54.950847', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEYNNF19zBoVS1kVFXKzKx3oHddUa7F+gm1LxTF1S3xseqpR6Yg+Bh5hKPEE4jRdxE4jzk2CIvKkJSIIVI7gHc='} 2024-04-27T02:32:52.226 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 01:52:54.951755', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU/fT/00CE2Kkjl5/ikJuKfY8klddqJtX8hdbl4FL8HUrO/4pHyw+xVuOYXLvAGr4fT58n7r28aEwZdSK7r3zc='} 2024-04-27T02:32:52.255 INFO:teuthology.orchestra.console.smithi008:Power off 2024-04-27T02:32:52.255 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T02:32:52.279 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-04-27T02:32:52.279 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:32:52.293 INFO:teuthology.orchestra.console.smithi077:Power off 2024-04-27T02:32:52.293 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T02:32:52.302 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-04-27T02:32:52.315 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-04-27T02:32:52.315 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:32:52.337 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-27T02:32:56.303 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:32:56.339 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:32:56.414 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-04-27T02:32:56.448 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-04-27T02:33:00.419 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:33:00.449 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T02:33:00.529 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-04-27T02:33:00.557 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-04-27T02:33:00.631 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-04-27T02:33:00.658 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-04-27T02:33:00.762 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2043.771292924881 failure_reason: '"2024-04-27T02:14:44.978233+0000 mon.smithi008 (mon.0) 882 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-27T02:33:00.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T02:33:00.845 INFO:teuthology.run:FAIL