2024-04-27T00:54:02.033 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T00:54:02.033 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445 --name teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/orig.config.yaml 2024-04-27T00:54:02.071 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T00:54:02.154 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675445' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 4854 sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a sleep_before_teardown: 0 subset: 40/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDluEo2n692/ghh2W9N94ppXOknaEDw7tvLcNNAkzp7idm2zlPynFhTEVQA8qwGuTZQCBDnFXUDi8O7FWFpu0Zk= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH++j15LfKWyud1/PZY3bWcv3bovYDD196xqx7r8LBDV9pXC1tWjU8KO73ZIdT9FT5blwPmDcPrBp4MmxCLFWmM= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-27T00:54:02.154 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-27T00:54:02.155 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-27T00:54:02.155 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T00:54:02.156 INFO:teuthology.task.internal:Checking packages... 2024-04-27T00:54:02.180 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a' 2024-04-27T00:54:02.180 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T00:54:02.180 INFO:teuthology.packaging:ref: None 2024-04-27T00:54:02.180 INFO:teuthology.packaging:tag: None 2024-04-27T00:54:02.180 INFO:teuthology.packaging:branch: squid 2024-04-27T00:54:02.180 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T00:54:02.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-27T00:54:02.629 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447.gb22e2ebd 2024-04-27T00:54:02.631 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T00:54:02.638 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T00:54:02.638 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T00:54:02.644 INFO:teuthology.task.internal:Saving configuration 2024-04-27T00:54:02.658 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T00:54:02.664 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T00:54:02.689 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:30.241081', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDluEo2n692/ghh2W9N94ppXOknaEDw7tvLcNNAkzp7idm2zlPynFhTEVQA8qwGuTZQCBDnFXUDi8O7FWFpu0Zk='} 2024-04-27T00:54:02.707 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:30.239713', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH++j15LfKWyud1/PZY3bWcv3bovYDD196xqx7r8LBDV9pXC1tWjU8KO73ZIdT9FT5blwPmDcPrBp4MmxCLFWmM='} 2024-04-27T00:54:02.707 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T00:54:02.714 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-27T00:54:02.714 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-27T00:54:02.714 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T00:54:02.790 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c31edff70>, signals=[15]) 2024-04-27T00:54:02.790 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T00:54:02.796 INFO:teuthology.task.internal:Opening connections... 2024-04-27T00:54:02.796 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T00:54:02.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:54:02.874 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-04-27T00:54:02.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:54:02.948 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T00:54:02.954 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-04-27T00:54:02.972 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-04-27T00:54:02.972 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-04-27T00:54:03.029 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-04-27T00:54:03.029 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-04-27T00:54:03.029 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T00:54:03.030 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T00:54:03.031 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-04-27T00:54:03.055 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-04-27T00:54:03.073 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-04-27T00:54:03.073 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-04-27T00:54:03.130 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-04-27T00:54:03.130 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T00:54:03.131 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T00:54:03.131 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-04-27T00:54:03.154 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T00:54:03.163 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T00:54:03.168 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T00:54:03.169 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-04-27T00:54:03.171 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-04-27T00:54:03.187 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T00:54:03.193 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T00:54:03.193 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-04-27T00:54:03.228 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-04-27T00:54:03.315 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T00:54:03.371 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T00:54:03.391 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T00:54:03.391 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T00:54:03.392 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T00:54:03.392 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T00:54:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:54:03.409 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-04-27T00:54:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:54:03.411 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-04-27T00:54:03.466 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-437.el9.x86_64 2024-04-27T00:54:03.466 INFO:teuthology.task.kernel:Running kernel on smithi135: 5.14.0-437.el9.x86_64 2024-04-27T00:54:03.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2024-04-27T00:54:03.467 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-437.el9.x86_64 2024-04-27T00:54:03.467 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-437.el9.x86_64 2024-04-27T00:54:03.467 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-04-27T00:54:05.891 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-27T00:54:06.516 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-04-27T00:54:08.240 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-27T00:54:13.241 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.2 MB 00:01 2024-04-27T00:54:14.684 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-27T00:54:15.871 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-27T00:54:16.865 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-04-27T00:54:20.894 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-27T00:54:20.965 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-27T00:54:22.349 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-04-27T00:54:22.792 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T00:54:22.792 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T00:54:22.882 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-04-27T00:54:22.883 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-04-27T00:54:22.883 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-04-27T00:54:22.995 DEBUG:teuthology.orchestra.run.smithi135:> echo no | sudo yum reinstall kernel || true 2024-04-27T00:54:23.497 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 12:54:20 AM UTC. 2024-04-27T00:54:23.617 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout:Reinstalling: 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-04-27T00:54:23.631 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:23.632 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:23.632 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 255 k 2024-04-27T00:54:23.632 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 0 2024-04-27T00:54:23.632 INFO:teuthology.orchestra.run.smithi135.stderr:Operation aborted. 2024-04-27T00:54:23.691 INFO:teuthology.orchestra.run.smithi135.stdout:Is this ok [y/N]: 2024-04-27T00:54:23.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum reinstall -y kernel || true 2024-04-27T00:54:24.184 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:04 ago on Sat 27 Apr 2024 12:54:20 AM UTC. 2024-04-27T00:54:24.299 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout:Reinstalling: 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-04-27T00:54:24.300 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-04-27T00:54:24.301 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:24.301 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 255 k 2024-04-27T00:54:24.301 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 0 2024-04-27T00:54:24.301 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-04-27T00:54:25.313 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-437.el9.x86_64.rpm 269 kB/s | 255 kB 00:00 2024-04-27T00:54:25.314 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-04-27T00:54:25.314 INFO:teuthology.orchestra.run.smithi135.stdout:Total 251 kB/s | 255 kB 00:01 2024-04-27T00:54:25.330 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-04-27T00:54:25.529 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-04-27T00:54:25.529 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-04-27T00:54:25.534 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-04-27T00:54:25.534 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-04-27T00:54:25.672 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-04-27T00:54:25.748 INFO:teuthology.orchestra.run.smithi135.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:54:25.748 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:26.033 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 21 MB 00:03 2024-04-27T00:54:26.484 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:26.484 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:54:27.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:27.168 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:27.168 INFO:teuthology.orchestra.run.smithi135.stdout:Reinstalled: 2024-04-27T00:54:27.169 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T00:54:27.169 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T00:54:27.169 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-04-27T00:54:27.350 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T00:54:27.398 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T00:54:27.399 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T00:54:27.399 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T00:54:27.399 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-27T00:54:27.399 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T00:54:27.425 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2024-04-27T00:54:27.452 DEBUG:teuthology.parallel:result is None 2024-04-27T00:54:31.132 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-27T00:54:32.980 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T00:54:32.980 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T00:54:33.066 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-27T00:54:33.067 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-04-27T00:54:33.067 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-27T00:54:33.181 DEBUG:teuthology.orchestra.run.smithi104:> echo no | sudo yum reinstall kernel || true 2024-04-27T00:54:33.716 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 12:54:31 AM UTC. 2024-04-27T00:54:33.836 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-27T00:54:33.848 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:33.848 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-04-27T00:54:33.848 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:33.848 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2024-04-27T00:54:33.848 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 255 k 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2024-04-27T00:54:33.849 INFO:teuthology.orchestra.run.smithi104.stderr:Operation aborted. 2024-04-27T00:54:33.907 INFO:teuthology.orchestra.run.smithi104.stdout:Is this ok [y/N]: 2024-04-27T00:54:33.907 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum reinstall -y kernel || true 2024-04-27T00:54:34.397 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 12:54:31 AM UTC. 2024-04-27T00:54:34.513 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-27T00:54:34.514 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-04-27T00:54:34.514 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:34.514 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2024-04-27T00:54:34.514 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 255 k 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2024-04-27T00:54:34.515 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-27T00:54:34.801 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-27T00:54:34.801 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-27T00:54:34.801 INFO:teuthology.orchestra.run.smithi104.stdout:Total 891 kB/s | 255 kB 00:00 2024-04-27T00:54:34.817 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-27T00:54:35.008 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-27T00:54:35.008 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-27T00:54:35.013 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-27T00:54:35.014 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-27T00:54:35.150 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-27T00:54:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:54:35.227 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:35.847 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:54:36.551 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:54:36.552 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:36.552 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalled: 2024-04-27T00:54:36.552 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T00:54:36.552 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T00:54:36.552 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-27T00:54:36.741 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T00:54:36.788 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T00:54:36.789 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T00:54:36.789 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T00:54:36.789 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-27T00:54:36.789 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T00:54:36.818 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-04-27T00:54:36.837 DEBUG:teuthology.parallel:result is None 2024-04-27T00:54:36.837 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T00:54:36.857 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T00:54:36.857 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T00:54:36.880 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T00:54:36.897 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T00:54:36.904 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T00:54:36.912 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T00:54:36.913 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T00:54:36.935 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T00:54:36.986 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T00:54:36.993 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T00:54:36.993 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T00:54:37.015 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T00:54:37.038 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:54:37.048 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:54:37.052 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:54:37.063 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:54:37.065 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T00:54:37.075 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T00:54:37.076 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T00:54:37.092 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T00:54:37.133 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T00:54:37.144 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T00:54:37.145 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T00:54:37.156 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T00:54:37.190 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:54:37.252 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:54:37.295 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:54:37.378 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:54:37.415 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:54:37.415 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T00:54:37.478 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:54:37.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:54:37.577 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:54:37.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:54:37.703 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T00:54:37.704 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T00:54:37.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-04-27T00:54:37.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-04-27T00:54:37.811 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:54:37.852 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:54:38.207 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T00:54:38.214 INFO:teuthology.task.internal:Starting timer... 2024-04-27T00:54:38.214 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T00:54:38.252 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T00:54:38.268 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-27T00:54:38.269 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-04-27T00:54:38.337 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-04-27T00:54:38.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-04-27T00:54:38.411 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-04-27T00:54:38.413 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T00:54:38.413 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-04-27T00:54:38.444 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-04-27T00:54:38.444 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-04-27T00:54:38.464 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-04-27T00:54:38.465 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2024-04-27T00:54:38.465 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:54:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:54:38.509 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:54:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:54:38.533 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T00:54:38.534 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-04-27T00:54:38.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-04-27T00:54:38.598 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T00:54:38.607 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-27T00:54:38.612 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-27T00:54:38.752 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T00:54:38.777 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T00:54:38.779 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T00:58:12.079 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2024-04-27T00:58:12.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-27T00:58:12.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:58:12.154 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-27T00:58:12.241 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-27T00:58:12.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-04-27T00:58:12.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:58:12.326 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-04-27T00:58:12.402 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-04-27T00:58:12.403 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T00:58:12.418 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T00:58:12.419 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T00:58:12.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:58:12.421 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T00:58:12.421 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:58:12.447 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T00:58:12.461 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T00:58:12.471 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T00:58:12.483 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-04-27T00:58:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T00:58:12.492 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-04-27T00:58:12.505 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T00:58:12.508 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-04-27T00:58:12.518 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T00:58:12.519 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-04-27T00:58:12.532 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T00:58:12.545 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T00:58:12.574 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.578 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.603 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:58:12.606 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:58:12.607 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-04-27T00:58:12.607 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.607 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.607 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.607 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:58:12.607 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-27T00:58:12.730 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T00:58:12.730 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-27T00:58:12.731 INFO:tasks.cephadm:Cluster fsid is 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:12.731 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-27T00:58:12.731 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-27T00:58:12.731 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi104': '172.21.15.104', 'mon.smithi135': '172.21.15.135'} 2024-04-27T00:58:12.731 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-27T00:58:12.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-04-27T00:58:12.756 DEBUG:teuthology.orchestra.run.smithi135:> sudo hostname $(hostname -s) 2024-04-27T00:58:12.780 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-27T00:58:12.781 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.186 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 27 00:58 /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.187 DEBUG:teuthology.orchestra.run.smithi135:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.516 INFO:teuthology.orchestra.run.smithi135.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 27 00:58 /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.516 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.533 DEBUG:teuthology.orchestra.run.smithi135:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-27T00:58:13.560 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-27T00:58:13.560 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-27T00:58:13.576 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-27T00:58:15.121 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T00:58:15.237 INFO:teuthology.orchestra.run.smithi135.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: "repo_digests": [ 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-04-27T00:58:33.729 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-27T00:58:35.861 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T00:58:35.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-04-27T00:58:35.932 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2024-04-27T00:58:35.963 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-04-27T00:58:35.987 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /etc/ceph 2024-04-27T00:58:36.031 INFO:tasks.cephadm:Writing seed config... 2024-04-27T00:58:36.032 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-27T00:58:36.032 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-27T00:58:36.033 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-27T00:58:36.033 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:58:36.033 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-27T00:58:36.048 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 38f5b578-0431-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-27T00:58:36.049 DEBUG:teuthology.orchestra.run.smithi104:mon.smithi104> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service 2024-04-27T00:58:36.092 INFO:tasks.cephadm:Bootstrapping... 2024-04-27T00:58:36.092 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 38f5b578-0431-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.104 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:58:36.281 INFO:teuthology.orchestra.run.smithi104.stderr:-------------------------------------------------------------------------------- 2024-04-27T00:58:36.281 INFO:teuthology.orchestra.run.smithi104.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '38f5b578-0431-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.104', '--skip-admin-label'] 2024-04-27T00:58:36.303 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.9.4-dev 2024-04-27T00:58:36.306 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying podman|docker is present... 2024-04-27T00:58:36.331 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.9.4-dev 2024-04-27T00:58:36.333 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying lvm2 is present... 2024-04-27T00:58:36.333 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying time synchronization is in place... 2024-04-27T00:58:36.340 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T00:58:36.346 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:58:36.353 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: enabled 2024-04-27T00:58:36.358 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: active 2024-04-27T00:58:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:Unit chronyd.service is enabled and running 2024-04-27T00:58:36.359 INFO:teuthology.orchestra.run.smithi104.stderr:Repeating the final host check... 2024-04-27T00:58:36.379 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: 4.9.4-dev 2024-04-27T00:58:36.381 INFO:teuthology.orchestra.run.smithi104.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-27T00:58:36.381 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl is present 2024-04-27T00:58:36.381 INFO:teuthology.orchestra.run.smithi104.stderr:lvcreate is present 2024-04-27T00:58:36.387 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T00:58:36.393 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:58:36.400 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: enabled 2024-04-27T00:58:36.405 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: active 2024-04-27T00:58:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:Unit chronyd.service is enabled and running 2024-04-27T00:58:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:Host looks OK 2024-04-27T00:58:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:Cluster fsid: 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:Acquiring lock 140297728939296 on /run/cephadm/38f5b578-0431-11ef-bc93-c7b262605968.lock 2024-04-27T00:58:36.406 INFO:teuthology.orchestra.run.smithi104.stderr:Lock 140297728939296 acquired on /run/cephadm/38f5b578-0431-11ef-bc93-c7b262605968.lock 2024-04-27T00:58:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying IP 172.21.15.104 port 3300 ... 2024-04-27T00:58:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying IP 172.21.15.104 port 6789 ... 2024-04-27T00:58:36.407 INFO:teuthology.orchestra.run.smithi104.stderr:Base mon IP is 172.21.15.104, final addrv is [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-04-27T00:58:36.409 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.104 metric 100 2024-04-27T00:58:36.410 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.104 metric 100 2024-04-27T00:58:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-27T00:58:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-27T00:58:36.412 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: inet6 fe80::2c3:e712:e84:c85b/64 scope link noprefixroute 2024-04-27T00:58:36.415 INFO:teuthology.orchestra.run.smithi104.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-27T00:58:36.416 INFO:teuthology.orchestra.run.smithi104.stderr:Mon IP `172.21.15.104` is in CIDR network `172.21.0.0/20` 2024-04-27T00:58:36.416 INFO:teuthology.orchestra.run.smithi104.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-27T00:58:36.417 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-27T00:58:36.520 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-27T00:58:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Getting image source signatures 2024-04-27T00:58:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-27T00:58:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-27T00:58:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-27T00:58:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-27T00:58:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-27T00:58:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-27T00:58:38.841 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: Writing manifest to image destination 2024-04-27T00:58:40.199 INFO:teuthology.orchestra.run.smithi104.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-27T00:58:43.982 INFO:teuthology.orchestra.run.smithi104.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-27T00:58:46.397 INFO:teuthology.orchestra.run.smithi104.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-27T00:58:46.398 INFO:teuthology.orchestra.run.smithi104.stderr:Extracting ceph user uid/gid from container image... 2024-04-27T00:58:46.852 INFO:teuthology.orchestra.run.smithi104.stderr:stat: 167 167 2024-04-27T00:58:47.443 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial keys... 2024-04-27T00:58:47.999 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQDHTSxmpnk/NxAAKvTIRZWH1XJZRio0WjhSWw== 2024-04-27T00:58:48.896 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQDITSxmBfN+MBAAdZcAYFa8yiThUxW7OB7E5g== 2024-04-27T00:58:49.760 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph-authtool: AQDJTSxmO/FEKBAAdizo7ZAVCGmpoDwC7ze3Zg== 2024-04-27T00:58:50.375 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial monmap... 2024-04-27T00:58:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T00:58:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-27T00:58:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:50.773 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T00:58:51.971 INFO:teuthology.orchestra.run.smithi104.stderr:monmaptool for smithi104 [v2:172.21.15.104:3300,v1:172.21.15.104:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T00:58:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:setting min_mon_release = octopus 2024-04-27T00:58:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: set fsid to 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T00:58:51.972 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:58:51.972 INFO:teuthology.orchestra.run.smithi104.stderr:Creating mon... 2024-04-27T00:58:53.775 INFO:teuthology.orchestra.run.smithi104.stderr:create mon.smithi104 on 2024-04-27T00:58:53.991 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-27T00:58:54.137 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target. 2024-04-27T00:58:54.137 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target. 2024-04-27T00:58:54.424 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to reset failed state of unit ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service: Unit ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service not loaded. 2024-04-27T00:58:54.432 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target.wants/ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service → /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968@.service. 2024-04-27T00:58:55.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:55 smithi104 podman[21449]: 2024-04-27 00:58:55.039626655 +0000 UTC m=+0.334166814 container start 4b453e092bf432cd2cb0faa2272676a3057846489b68136b49afbd19ff8ae35f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.19.8, release=754) 2024-04-27T00:58:55.148 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-04-27T00:58:55.154 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-04-27T00:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to enable service . firewalld.service is not available 2024-04-27T00:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mon to start... 2024-04-27T00:58:55.155 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mon... 2024-04-27T00:58:55.369 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:55 smithi104 ceph-mon[21468]: mkfs 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:55.369 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:55 smithi104 ceph-mon[21468]: mon.smithi104 is new leader, mons smithi104 in quorum (ranks 0) 2024-04-27T00:58:55.369 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:55 smithi104 bash[21449]: 4b453e092bf432cd2cb0faa2272676a3057846489b68136b49afbd19ff8ae35f 2024-04-27T00:58:55.369 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:55 smithi104 systemd[1]: Started Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T00:58:55.635 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: cluster: 2024-04-27T00:58:55.635 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: id: 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: services: 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi104 (age 0.557768s) 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: data: 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: pgs: 2024-04-27T00:58:55.636 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:mon is available 2024-04-27T00:58:56.169 INFO:teuthology.orchestra.run.smithi104.stderr:Assimilating anything we can from ceph.conf... 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: mon.smithi104 is new leader, mons smithi104 in quorum (ranks 0) 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: monmap e1: 1 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: fsmap 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: mgrmap e1: no daemons active 2024-04-27T00:58:56.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:56 smithi104 ceph-mon[21468]: from='client.? 172.21.15.104:0/2125666971' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [global] 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: fsid = 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.104:3300,v1:172.21.15.104:6789] 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [mgr] 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-27T00:58:56.699 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:58:56.700 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [osd] 2024-04-27T00:58:56.700 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-27T00:58:56.700 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-27T00:58:57.175 INFO:teuthology.orchestra.run.smithi104.stderr:Generating new minimal ceph.conf... 2024-04-27T00:58:57.399 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:57 smithi104 ceph-mon[21468]: from='client.? 172.21.15.104:0/2083243061' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T00:58:57.399 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:57 smithi104 ceph-mon[21468]: from='client.? 172.21.15.104:0/2083243061' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-27T00:58:58.256 INFO:teuthology.orchestra.run.smithi104.stderr:Restarting the monitor... 2024-04-27T00:58:58.424 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 ceph-mon[21468]: from='client.? 172.21.15.104:0/3051534529' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:58:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T00:58:58.425 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 bash[21686]: Error: no container with name or ID "ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon.smithi104" found: no such container 2024-04-27T00:58:58.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[21464]: 2024-04-27T00:58:58.673+0000 7f7b66e33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:58:58.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[21464]: 2024-04-27T00:58:58.673+0000 7f7b66e33700 -1 mon.smithi104@0(leader) e1 *** Got Signal Terminated *** 2024-04-27T00:58:58.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:58 smithi104 podman[21697]: 2024-04-27 00:58:58.879253706 +0000 UTC m=+0.564764015 container died 4b453e092bf432cd2cb0faa2272676a3057846489b68136b49afbd19ff8ae35f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, RELEASE=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.buildah.version=1.19.8, name=centos-stream) 2024-04-27T00:58:59.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21697]: 2024-04-27 00:58:59.173370027 +0000 UTC m=+0.858880360 container cleanup 4b453e092bf432cd2cb0faa2272676a3057846489b68136b49afbd19ff8ae35f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.expose-services=, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vendor=Red Hat, Inc., io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-27T00:58:59.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 bash[21697]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104 2024-04-27T00:58:59.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 bash[21723]: Error: no container with name or ID "ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon.smithi104" found: no such container 2024-04-27T00:58:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21711]: 2024-04-27 00:58:59.386350577 +0000 UTC m=+0.503069681 container remove 4b453e092bf432cd2cb0faa2272676a3057846489b68136b49afbd19ff8ae35f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, ceph=True, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_CLEAN=True, distribution-scope=public, version=8, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux ) 2024-04-27T00:58:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service: Deactivated successfully. 2024-04-27T00:58:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 systemd[1]: Stopped Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T00:58:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 systemd[1]: Starting Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T00:58:59.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21792]: 2024-04-27 00:58:59.616470677 +0000 UTC m=+0.112142689 container create 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-27T00:58:59.675 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21792]: 2024-04-27 00:58:59.522872659 +0000 UTC m=+0.018544680 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-27T00:58:59.896 INFO:teuthology.orchestra.run.smithi104.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-27T00:59:00.020 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21792]: 2024-04-27 00:58:59.81035597 +0000 UTC m=+0.306027982 container init 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, io.openshift.expose-services=, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, version=8) 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 podman[21792]: 2024-04-27 00:58:59.813251202 +0000 UTC m=+0.308923221 container start 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, ceph=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, vcs-type=git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, distribution-scope=public, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: pidfile_write: ignore empty --pid-file 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: load: jerasure load: lrc 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: RocksDB version: 6.15.5 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Compile date Apr 18 2022 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: DB SUMMARY 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: DB Session ID: ZBCFW5QLSAB3XCGRVQP1 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: CURRENT file: CURRENT 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-27T00:59:00.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 1, files: 000008.sst 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 000010.log size: 82741 ; 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.error_if_exists: 0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.create_if_missing: 0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.paranoid_checks: 1 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.env: 0x560b890c6860 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.fs: Posix File System 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.info_log: 0x560b8af0bfa0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.statistics: (nil) 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.use_fsync: 0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_log_file_size: 0 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T00:59:00.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_fallocate: 1 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.use_direct_reads: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.db_log_dir: 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi104/store.db 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T00:59:00.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.write_buffer_manager: 0x560b8affc2d0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.unordered_write: 0 2024-04-27T00:59:00.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.row_cache: None 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.wal_filter: None 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.preserve_deletes: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.two_write_queues: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.atomic_flush: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T00:59:00.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.log_readahead_size: 0 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_background_jobs: 2 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_background_compactions: -1 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_subcompactions: 1 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T00:59:00.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T00:59:00.027 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_open_files: -1 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_background_flushes: -1 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Compression algorithms supported: 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kZSTD supported: 0 2024-04-27T00:59:00.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kXpressCompression supported: 0 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kLZ4Compression supported: 1 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kBZip2Compression supported: 0 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kZlibCompression supported: 1 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: kSnappyCompression supported: 1 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000009 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.merge_operator: 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_filter: None 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T00:59:00.029 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b8aed7d00) 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_type: 0 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_index_type: 0 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_shortening: 1 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: hash_index_allow_collision: 1 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: checksum: 1 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: no_block_cache: 0 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache: 0x560b8af42170 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-27T00:59:00.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_options: 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: capacity : 536870912 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_shard_bits : 4 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: strict_capacity_limit : 0 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_compressed: (nil) 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: persistent_cache: (nil) 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size: 4096 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size_deviation: 10 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_restart_interval: 16 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_block_restart_interval: 1 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_block_size: 4096 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: partition_filters: 0 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: use_delta_encoding: 1 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-27T00:59:00.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: whole_key_filtering: 1 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: verify_compression: 0 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: format_version: 4 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_index_compression: 1 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_align: 0 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression: NoCompression 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.num_levels: 7 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T00:59:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T00:59:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T00:59:00.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.arena_block_size: 4194304 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-27T00:59:00.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.table_properties_collectors: 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.inplace_update_support: 0 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T00:59:00.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.bloom_locality: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.max_successive_merges: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.ttl: 2592000 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.enable_blob_files: false 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.min_blob_size: 0 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T00:59:00.037 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179539839247, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179539840225, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79423, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77687, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71996, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714179539, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "b20ce235-7031-44e0-8824-3deb5867ac6a", "db_session_id": "ZBCFW5QLSAB3XCGRVQP1"}} 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179539840653, "job": 1, "event": "recovery_finished"} 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi104/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560b8af28a80 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: DB pointer 0x560b8af38000 2024-04-27T00:59:00.038 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** DB Stats ** 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 2/0 79.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.039 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 2/0 79.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 18.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 18.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T00:59:00.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 2/0 79.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 2/0 79.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T00:59:00.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.7 0.00 0.00 1 0.001 0 0 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 18.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: starting mon.smithi104 rank 0 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi104 fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???) e1 preinit fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:00.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).mds e1 new map 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).mds e1 print_map 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: e1 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: legacy client fscid: -1 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: No filesystems configured 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 bash[21792]: 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a 2024-04-27T00:59:00.043 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 systemd[1]: Started Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T00:59:00.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mon.smithi104 is new leader, mons smithi104 in quorum (ranks 0) 2024-04-27T00:59:00.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: monmap e1: 1 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} 2024-04-27T00:59:00.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: fsmap 2024-04-27T00:59:00.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T00:59:00.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:58:59 smithi104 ceph-mon[21811]: mgrmap e1: no daemons active 2024-04-27T00:59:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-27T00:59:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:59:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:Creating mgr... 2024-04-27T00:59:01.027 INFO:teuthology.orchestra.run.smithi104.stderr:Verifying port 9283 ... 2024-04-27T00:59:01.191 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Failed to reset failed state of unit ceph-38f5b578-0431-11ef-bc93-c7b262605968@mgr.smithi104.dpemzn.service: Unit ceph-38f5b578-0431-11ef-bc93-c7b262605968@mgr.smithi104.dpemzn.service not loaded. 2024-04-27T00:59:01.198 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: Created symlink /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968.target.wants/ceph-38f5b578-0431-11ef-bc93-c7b262605968@mgr.smithi104.dpemzn.service → /etc/systemd/system/ceph-38f5b578-0431-11ef-bc93-c7b262605968@.service. 2024-04-27T00:59:01.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:01 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1806651283' entity='client.admin' 2024-04-27T00:59:01.859 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-04-27T00:59:01.866 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:59:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-04-27T00:59:01.867 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to enable service . firewalld.service is not available 2024-04-27T00:59:01.875 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-04-27T00:59:01.882 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:59:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-04-27T00:59:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-27T00:59:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr to start... 2024-04-27T00:59:01.883 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr... 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "38f5b578-0431-11ef-bc93-c7b262605968", 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-04-27T00:59:02.469 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T00:59:02.470 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T00:59:02.471 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T00:59:02.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T00:59:02.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T00:59:02.474 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T00:59:02.475 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": false, 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "nfs", 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-04-27T00:58:55.078798+0000", 2024-04-27T00:59:02.476 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:02.477 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:02.477 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T00:59:02.477 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:02.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:02 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/668660675' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T00:59:03.389 INFO:teuthology.orchestra.run.smithi104.stderr:mgr not available, waiting (1/15)... 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "38f5b578-0431-11ef-bc93-c7b262605968", 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:05.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T00:59:05.898 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T00:59:05.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T00:59:05.902 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T00:59:05.903 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": false, 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "nfs", 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-04-27T00:58:55.078798+0000", 2024-04-27T00:59:05.904 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:05.905 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:05.905 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T00:59:05.905 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:06.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:05 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1081862682' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T00:59:06.493 INFO:teuthology.orchestra.run.smithi104.stderr:mgr not available, waiting (2/15)... 2024-04-27T00:59:08.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: Activating manager daemon smithi104.dpemzn 2024-04-27T00:59:08.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: mgrmap e2: smithi104.dpemzn(active, starting, since 0.0028872s) 2024-04-27T00:59:08.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T00:59:08.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: Manager daemon smithi104.dpemzn is now available 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:08.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:07 smithi104 ceph-mon[21811]: from='mgr.14100 172.21.15.104:0/581894465' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsid": "38f5b578-0431-11ef-bc93-c7b262605968", 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "health": { 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "checks": {}, 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mutes": [] 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.149 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum": [ 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 0 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "smithi104" 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "monmap": { 2024-04-27T00:59:09.150 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osdmap": { 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-27T00:59:09.151 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-27T00:59:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-27T00:59:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-27T00:59:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.156 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgmap": { 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "fsmap": { 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-27T00:59:09.157 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modules": [ 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "iostat", 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "nfs", 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "restful" 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ], 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.158 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "servicemap": { 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "modified": "2024-04-27T00:58:55.078798+0000", 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "services": {} 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: }, 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-27T00:59:09.159 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:09.749 INFO:teuthology.orchestra.run.smithi104.stderr:mgr is available 2024-04-27T00:59:09.871 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:09 smithi104 ceph-mon[21811]: mgrmap e3: smithi104.dpemzn(active, since 1.00784s) 2024-04-27T00:59:09.871 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:09 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2982331175' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [global] 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: fsid = 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-27T00:59:10.304 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [mgr] 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: [osd] 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-27T00:59:10.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-27T00:59:10.878 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:10 smithi104 ceph-mon[21811]: mgrmap e4: smithi104.dpemzn(active, since 2s) 2024-04-27T00:59:10.879 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:10 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1257693894' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T00:59:10.881 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling cephadm module... 2024-04-27T00:59:12.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:11 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2419232711' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-27T00:59:13.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:12 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2419232711' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-27T00:59:13.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:12 smithi104 ceph-mon[21811]: mgrmap e5: smithi104.dpemzn(active, since 4s) 2024-04-27T00:59:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-27T00:59:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-04-27T00:59:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "active_name": "smithi104.dpemzn", 2024-04-27T00:59:13.170 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-27T00:59:13.171 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:13.718 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for the mgr to restart... 2024-04-27T00:59:13.718 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr epoch 5... 2024-04-27T00:59:14.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:13 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3430850050' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T00:59:18.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:17 smithi104 ceph-mon[21811]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T00:59:18.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:17 smithi104 ceph-mon[21811]: Activating manager daemon smithi104.dpemzn 2024-04-27T00:59:18.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:17 smithi104 ceph-mon[21811]: osdmap e2: 0 total, 0 up, 0 in 2024-04-27T00:59:18.888 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:18.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-27T00:59:18.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "initialized": true 2024-04-27T00:59:18.889 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:19.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: mgrmap e6: smithi104.dpemzn(active, starting, since 0.0524534s) 2024-04-27T00:59:19.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T00:59:19.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T00:59:19.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T00:59:19.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: Manager daemon smithi104.dpemzn is now available 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:19.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:18 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:19.490 INFO:teuthology.orchestra.run.smithi104.stderr:mgr epoch 5 is available 2024-04-27T00:59:19.491 INFO:teuthology.orchestra.run.smithi104.stderr:Setting orchestrator backend to cephadm... 2024-04-27T00:59:20.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:19 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:18] ENGINE Bus STARTING 2024-04-27T00:59:20.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:19 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:18] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T00:59:20.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:19 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:18] ENGINE Bus STARTED 2024-04-27T00:59:20.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:19 smithi104 ceph-mon[21811]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T00:59:20.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:19 smithi104 ceph-mon[21811]: mgrmap e7: smithi104.dpemzn(active, since 1.06055s) 2024-04-27T00:59:21.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:20 smithi104 ceph-mon[21811]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T00:59:21.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:20 smithi104 ceph-mon[21811]: mgrmap e8: smithi104.dpemzn(active, since 2s) 2024-04-27T00:59:21.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:20 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:21.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:20 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: value unchanged 2024-04-27T00:59:21.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:21 smithi104 ceph-mon[21811]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:22.001 INFO:teuthology.orchestra.run.smithi104.stderr:Generating ssh key... 2024-04-27T00:59:23.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:22 smithi104 ceph-mon[21811]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:23.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:22 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:23.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:22 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:23.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:23 smithi104 ceph-mon[21811]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:23.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:23 smithi104 ceph-mon[21811]: Generating ssh key... 2024-04-27T00:59:23.995 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLj/Bdr3knHFauw/YDQBWvZQKBpS1UaDwCv05bBXYWY8MXLE/D3/JyqtHXnP+4gpoYax+f9M80It2DvZCsotK3nKSkRdZivnXjSWdCts3hviVzFg1ve8yQlyWEuuVhjr4TNhim0X7kRd25BW6gIjEVKsymazD0qvXzYaFjXVHDVmP+JaEQyFrJ2ajdwEMSiSvmnAZR6oJWCC4O1no1NEB6caWoiJPgJ/LlF03nzPKjh8N4vWMaVDdsT0skys1q3SY0wws75ymwDMEYWmCXkfGoWThg3koEJuzJFWE5yXTiso6gq3IGJqpnbthbgWXuxKTFrKBc3gzupSWRQueuzRUXaKvgBKu/QLoLPknR4qbIuoBBiX4/bD4QEcGua642L+i6jJR+wzaHkCw8BCpRkW0DZAM+phz0Jgvf6tfXm6gpKxUz7ef43Xtbj25wQ/QvJn1//yYbEtxtoiZK8qjeALK6wWNvKD8V5S2hxfUq2mIVeREqjgx0nQj9W2Rtp+C4hhM= ceph-38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:24.589 INFO:teuthology.orchestra.run.smithi104.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:59:24.589 INFO:teuthology.orchestra.run.smithi104.stderr:Adding key to root@localhost authorized_keys... 2024-04-27T00:59:24.590 INFO:teuthology.orchestra.run.smithi104.stderr:Adding host smithi104... 2024-04-27T00:59:26.074 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Added host 'smithi104' with addr '172.21.15.104' 2024-04-27T00:59:26.140 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:25 smithi104 ceph-mon[21811]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:26.662 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying mon service with default placement... 2024-04-27T00:59:26.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:26 smithi104 ceph-mon[21811]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "addr": "172.21.15.104", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:26.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:26 smithi104 ceph-mon[21811]: Deploying cephadm binary to smithi104 2024-04-27T00:59:26.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:26 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:26.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:26 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:27.299 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-27T00:59:27.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:27 smithi104 ceph-mon[21811]: Added host smithi104 2024-04-27T00:59:27.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:27 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:28.040 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying mgr service with default placement... 2024-04-27T00:59:28.673 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-27T00:59:28.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:28 smithi104 ceph-mon[21811]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:28.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:28 smithi104 ceph-mon[21811]: Saving service mon spec with placement count:5 2024-04-27T00:59:28.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:28 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:29.690 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying crash service with default placement... 2024-04-27T00:59:30.072 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:29 smithi104 ceph-mon[21811]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:30.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:29 smithi104 ceph-mon[21811]: Saving service mgr spec with placement count:2 2024-04-27T00:59:30.306 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-27T00:59:31.355 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying prometheus service with default placement... 2024-04-27T00:59:31.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:31 smithi104 ceph-mon[21811]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:31.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:31 smithi104 ceph-mon[21811]: Saving service crash spec with placement * 2024-04-27T00:59:31.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:31 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:31.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:31 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:31.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:31 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:32.017 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-27T00:59:32.685 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying grafana service with default placement... 2024-04-27T00:59:33.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:33 smithi104 ceph-mon[21811]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:33.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:33 smithi104 ceph-mon[21811]: Saving service prometheus spec with placement count:1 2024-04-27T00:59:33.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:33 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:33.305 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-27T00:59:33.934 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying node-exporter service with default placement... 2024-04-27T00:59:34.384 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:34 smithi104 ceph-mon[21811]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:34.384 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:34 smithi104 ceph-mon[21811]: Saving service grafana spec with placement count:1 2024-04-27T00:59:34.385 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:34 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:34.601 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-27T00:59:35.166 INFO:teuthology.orchestra.run.smithi104.stderr:Deploying alertmanager service with default placement... 2024-04-27T00:59:35.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:35 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:35.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:35 smithi104 ceph-mon[21811]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:35.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:35 smithi104 ceph-mon[21811]: Saving service node-exporter spec with placement * 2024-04-27T00:59:35.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:35 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:35.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:35 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-27T00:59:36.933 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:36 smithi104 ceph-mon[21811]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:36.933 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:36 smithi104 ceph-mon[21811]: Saving service alertmanager spec with placement count:1 2024-04-27T00:59:36.933 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:36 smithi104 ceph-mon[21811]: from='mgr.14118 172.21.15.104:0/1757791669' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:38.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:37 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/150481963' entity='client.admin' 2024-04-27T00:59:38.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:37 smithi104 ceph-mon[21811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T00:59:39.423 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling the dashboard module... 2024-04-27T00:59:39.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:39 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2698072148' entity='client.admin' 2024-04-27T00:59:41.153 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:40 smithi104 ceph-mon[21811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T00:59:41.153 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:40 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2133406084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-27T00:59:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:41 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2133406084' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-27T00:59:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:41 smithi104 ceph-mon[21811]: mgrmap e9: smithi104.dpemzn(active, since 23s) 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "available": true, 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "active_name": "smithi104.dpemzn", 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-27T00:59:42.244 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:42.846 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for the mgr to restart... 2024-04-27T00:59:42.846 INFO:teuthology.orchestra.run.smithi104.stderr:Waiting for mgr epoch 9... 2024-04-27T00:59:43.053 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:42 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1169528799' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T00:59:47.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: { 2024-04-27T00:59:47.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-27T00:59:47.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: "initialized": true 2024-04-27T00:59:47.897 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: } 2024-04-27T00:59:48.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: Activating manager daemon smithi104.dpemzn 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: osdmap e3: 0 total, 0 up, 0 in 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: mgrmap e10: smithi104.dpemzn(active, starting, since 0.0529574s) 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T00:59:48.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: Manager daemon smithi104.dpemzn is now available 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T00:59:48.176 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:47 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:48.621 INFO:teuthology.orchestra.run.smithi104.stderr:mgr epoch 9 is available 2024-04-27T00:59:48.621 INFO:teuthology.orchestra.run.smithi104.stderr:Generating a dashboard self-signed certificate... 2024-04-27T00:59:49.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:47] ENGINE Bus STARTING 2024-04-27T00:59:49.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:47] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T00:59:49.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: [27/Apr/2024:00:59:47] ENGINE Bus STARTED 2024-04-27T00:59:49.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T00:59:49.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: mgrmap e11: smithi104.dpemzn(active, since 1.05493s) 2024-04-27T00:59:49.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T00:59:49.174 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:48 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:49.389 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-27T00:59:49.994 INFO:teuthology.orchestra.run.smithi104.stderr:Creating initial admin user... 2024-04-27T00:59:50.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:50 smithi104 ceph-mon[21811]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:50.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:50 smithi104 ceph-mon[21811]: mgrmap e12: smithi104.dpemzn(active, since 2s) 2024-04-27T00:59:50.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:50 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:50.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:50 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:50.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:50 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:50.896 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Qs8h3imBNX7gRx85HHasdu6YS.eNKMhUvmocXLRtUsUMq5gBQoQTq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714179590, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-27T00:59:52.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:51 smithi104 ceph-mon[21811]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:59:52.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:51 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:52.269 INFO:teuthology.orchestra.run.smithi104.stderr:Fetching dashboard port number... 2024-04-27T00:59:52.872 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: 8443 2024-04-27T00:59:53.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:52 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/240871932' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-27T00:59:53.561 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: disabled 2024-04-27T00:59:53.580 INFO:teuthology.orchestra.run.smithi104.stderr:systemctl: inactive 2024-04-27T00:59:53.580 INFO:teuthology.orchestra.run.smithi104.stderr:firewalld.service is not enabled 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr:Ceph Dashboard is now available at: 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr: URL: https://smithi104:8443/ 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr: User: admin 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr: Password: j5kpl9kfee 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:53.581 INFO:teuthology.orchestra.run.smithi104.stderr:Enabling autotune for osd_memory_target 2024-04-27T00:59:54.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:53 smithi104 ceph-mon[21811]: mgrmap e13: smithi104.dpemzn(active, since 6s) 2024-04-27T00:59:55.679 INFO:teuthology.orchestra.run.smithi104.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-27T00:59:56.254 INFO:teuthology.orchestra.run.smithi104.stderr:You can access the Ceph CLI with: 2024-04-27T00:59:56.254 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: ceph telemetry on 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr:For more information see: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T00:59:56.255 INFO:teuthology.orchestra.run.smithi104.stderr:Bootstrap complete. 2024-04-27T00:59:56.287 INFO:tasks.cephadm:Fetching config... 2024-04-27T00:59:56.287 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:59:56.287 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-27T00:59:56.303 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-27T00:59:56.303 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:59:56.303 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-27T00:59:56.359 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-27T00:59:56.359 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:59:56.360 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/keyring of=/dev/stdout 2024-04-27T00:59:56.422 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-27T00:59:56.422 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T00:59:56.422 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-27T00:59:56.478 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-27T00:59:56.478 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLj/Bdr3knHFauw/YDQBWvZQKBpS1UaDwCv05bBXYWY8MXLE/D3/JyqtHXnP+4gpoYax+f9M80It2DvZCsotK3nKSkRdZivnXjSWdCts3hviVzFg1ve8yQlyWEuuVhjr4TNhim0X7kRd25BW6gIjEVKsymazD0qvXzYaFjXVHDVmP+JaEQyFrJ2ajdwEMSiSvmnAZR6oJWCC4O1no1NEB6caWoiJPgJ/LlF03nzPKjh8N4vWMaVDdsT0skys1q3SY0wws75ymwDMEYWmCXkfGoWThg3koEJuzJFWE5yXTiso6gq3IGJqpnbthbgWXuxKTFrKBc3gzupSWRQueuzRUXaKvgBKu/QLoLPknR4qbIuoBBiX4/bD4QEcGua642L+i6jJR+wzaHkCw8BCpRkW0DZAM+phz0Jgvf6tfXm6gpKxUz7ef43Xtbj25wQ/QvJn1//yYbEtxtoiZK8qjeALK6wWNvKD8V5S2hxfUq2mIVeREqjgx0nQj9W2Rtp+C4hhM= ceph-38f5b578-0431-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T00:59:56.550 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLj/Bdr3knHFauw/YDQBWvZQKBpS1UaDwCv05bBXYWY8MXLE/D3/JyqtHXnP+4gpoYax+f9M80It2DvZCsotK3nKSkRdZivnXjSWdCts3hviVzFg1ve8yQlyWEuuVhjr4TNhim0X7kRd25BW6gIjEVKsymazD0qvXzYaFjXVHDVmP+JaEQyFrJ2ajdwEMSiSvmnAZR6oJWCC4O1no1NEB6caWoiJPgJ/LlF03nzPKjh8N4vWMaVDdsT0skys1q3SY0wws75ymwDMEYWmCXkfGoWThg3koEJuzJFWE5yXTiso6gq3IGJqpnbthbgWXuxKTFrKBc3gzupSWRQueuzRUXaKvgBKu/QLoLPknR4qbIuoBBiX4/bD4QEcGua642L+i6jJR+wzaHkCw8BCpRkW0DZAM+phz0Jgvf6tfXm6gpKxUz7ef43Xtbj25wQ/QvJn1//yYbEtxtoiZK8qjeALK6wWNvKD8V5S2hxfUq2mIVeREqjgx0nQj9W2Rtp+C4hhM= ceph-38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:56.560 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLj/Bdr3knHFauw/YDQBWvZQKBpS1UaDwCv05bBXYWY8MXLE/D3/JyqtHXnP+4gpoYax+f9M80It2DvZCsotK3nKSkRdZivnXjSWdCts3hviVzFg1ve8yQlyWEuuVhjr4TNhim0X7kRd25BW6gIjEVKsymazD0qvXzYaFjXVHDVmP+JaEQyFrJ2ajdwEMSiSvmnAZR6oJWCC4O1no1NEB6caWoiJPgJ/LlF03nzPKjh8N4vWMaVDdsT0skys1q3SY0wws75ymwDMEYWmCXkfGoWThg3koEJuzJFWE5yXTiso6gq3IGJqpnbthbgWXuxKTFrKBc3gzupSWRQueuzRUXaKvgBKu/QLoLPknR4qbIuoBBiX4/bD4QEcGua642L+i6jJR+wzaHkCw8BCpRkW0DZAM+phz0Jgvf6tfXm6gpKxUz7ef43Xtbj25wQ/QvJn1//yYbEtxtoiZK8qjeALK6wWNvKD8V5S2hxfUq2mIVeREqjgx0nQj9W2Rtp+C4hhM= ceph-38f5b578-0431-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T00:59:56.602 INFO:teuthology.orchestra.run.smithi135.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLj/Bdr3knHFauw/YDQBWvZQKBpS1UaDwCv05bBXYWY8MXLE/D3/JyqtHXnP+4gpoYax+f9M80It2DvZCsotK3nKSkRdZivnXjSWdCts3hviVzFg1ve8yQlyWEuuVhjr4TNhim0X7kRd25BW6gIjEVKsymazD0qvXzYaFjXVHDVmP+JaEQyFrJ2ajdwEMSiSvmnAZR6oJWCC4O1no1NEB6caWoiJPgJ/LlF03nzPKjh8N4vWMaVDdsT0skys1q3SY0wws75ymwDMEYWmCXkfGoWThg3koEJuzJFWE5yXTiso6gq3IGJqpnbthbgWXuxKTFrKBc3gzupSWRQueuzRUXaKvgBKu/QLoLPknR4qbIuoBBiX4/bD4QEcGua642L+i6jJR+wzaHkCw8BCpRkW0DZAM+phz0Jgvf6tfXm6gpKxUz7ef43Xtbj25wQ/QvJn1//yYbEtxtoiZK8qjeALK6wWNvKD8V5S2hxfUq2mIVeREqjgx0nQj9W2Rtp+C4hhM= ceph-38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T00:59:56.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-27T00:59:56.917 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:56 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1929875829' entity='client.admin' 2024-04-27T00:59:59.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:59 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:59 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-27T00:59:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:59 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T00:59:59.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 00:59:59 smithi104 ceph-mon[21811]: Deploying daemon alertmanager.smithi104 on smithi104 2024-04-27T01:00:00.095 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-27T01:00:00.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-27T01:00:00.581 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:00 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1151643509' entity='client.admin' 2024-04-27T01:00:03.502 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi135 2024-04-27T01:00:03.502 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T01:00:03.502 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.conf 2024-04-27T01:00:03.520 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T01:00:03.521 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:00:03.578 INFO:tasks.cephadm:Adding host smithi135 to orchestrator... 2024-04-27T01:00:03.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch host add smithi135 2024-04-27T01:00:03.887 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:03 smithi104 ceph-mon[21811]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:00:03.887 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:03 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:06.894 INFO:teuthology.orchestra.run.smithi104.stdout:Added host 'smithi135' with addr '172.21.15.135' 2024-04-27T01:00:07.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-27T01:00:08.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:07 smithi104 ceph-mon[21811]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:00:08.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:07 smithi104 ceph-mon[21811]: Deploying cephadm binary to smithi135 2024-04-27T01:00:08.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:07 smithi104 ceph-mon[21811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:08.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:07 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:08.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:07 smithi104 ceph-mon[21811]: Added host smithi135 2024-04-27T01:00:09.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:09 smithi104 ceph-mon[21811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:12.148 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:00:12.149 INFO:teuthology.orchestra.run.smithi104.stdout:[{"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "172.21.15.135", "hostname": "smithi135", "labels": [], "status": ""}] 2024-04-27T01:00:12.149 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:11 smithi104 ceph-mon[21811]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:13.613 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-27T01:00:13.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-27T01:00:14.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:00:14.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:14.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:14.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:00:14.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T01:00:14.107 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:13 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:15.049 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:14 smithi104 ceph-mon[21811]: Deploying daemon crash.smithi104 on smithi104 2024-04-27T01:00:16.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:15 smithi104 ceph-mon[21811]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:16.903 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-04-27T01:00:17.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:16 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2400900315' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:00:17.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:16 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:17.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:16 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:17.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:16 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:17.535 INFO:tasks.cephadm:Adding mon.smithi104 on smithi104 2024-04-27T01:00:17.535 INFO:tasks.cephadm:Adding mon.smithi135 on smithi135 2024-04-27T01:00:17.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135' 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2400900315' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: Deploying daemon grafana.smithi104 on smithi104 2024-04-27T01:00:18.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:17 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:19.253 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled mon update... 2024-04-27T01:00:19.806 DEBUG:teuthology.orchestra.run.smithi135:mon.smithi135> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi135.service 2024-04-27T01:00:19.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:19.809 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:20.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:19 smithi104 ceph-mon[21811]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:20.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:19 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:21.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:20 smithi104 ceph-mon[21811]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:00:21.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:20 smithi104 ceph-mon[21811]: Saving service mon spec with placement smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135;count:2 2024-04-27T01:00:21.539 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:21.540 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:21.542 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:22.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:21 smithi104 ceph-mon[21811]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:22.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:21 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/1581502963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:23.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:23.122 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:24.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:23 smithi104 ceph-mon[21811]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:25.043 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:25.043 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:25.044 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:26.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:25 smithi104 ceph-mon[21811]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:26.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:25 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3561745819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:26.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:26.619 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:27.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:26 smithi104 ceph-mon[21811]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:28.504 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:28.504 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:28.506 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:28.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:28 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3726879282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:29.797 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:29 smithi104 ceph-mon[21811]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:29.798 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:29 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:30.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:30.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:30 smithi104 ceph-mon[21811]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-27T01:00:31.900 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:31.900 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:31.902 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:32.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:31 smithi104 ceph-mon[21811]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:33.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:32 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/146576176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:33.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:32 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:33.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:33.463 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:34.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:33 smithi104 ceph-mon[21811]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:35.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:34 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:35.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:34 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:00:35.221 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:35.221 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:35.224 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:36.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:35 smithi104 ceph-mon[21811]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:00:36.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:35 smithi104 ceph-mon[21811]: mgrmap e14: smithi104.dpemzn(active, since 48s) 2024-04-27T01:00:36.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:35 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/2179445630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:36.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:36.827 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:38.653 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:38.654 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:38.655 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:39.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:38 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3065238310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:40.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:40.244 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:41.125 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:41 smithi104 ceph-mon[21811]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T01:00:41.125 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:41 smithi104 ceph-mon[21811]: Activating manager daemon smithi104.dpemzn 2024-04-27T01:00:41.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:41 smithi104 ceph-mon[21811]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:00:42.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: mgrmap e15: smithi104.dpemzn(active, starting, since 0.0524888s) 2024-04-27T01:00:42.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:00:42.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:42.188 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:42.188 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:42.190 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:43.353 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: mgrmap e16: smithi104.dpemzn(active, since 1.0633s) 2024-04-27T01:00:43.353 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:43.353 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/2536111955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:43.353 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: [27/Apr/2024:01:00:42] ENGINE Bus STARTING 2024-04-27T01:00:43.354 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: [27/Apr/2024:01:00:42] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:00:43.354 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: [27/Apr/2024:01:00:42] ENGINE Bus STARTED 2024-04-27T01:00:43.354 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:43.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:43.764 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:44 smithi104 ceph-mon[21811]: mgrmap e17: smithi104.dpemzn(active, since 2s) 2024-04-27T01:00:45.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:45.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:45.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:00:45.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:45 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:00:45.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:46.502 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:46.502 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:46.504 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:46.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:46 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:00:47.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:47 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/910644364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:48.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:48.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:48.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:48.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:48.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:00:48.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:48.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:00:48.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T01:00:48.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:48.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:48 smithi104 ceph-mon[21811]: Deploying daemon crash.smithi135 on smithi135 2024-04-27T01:00:50.549 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:50.549 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:50.551 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:51.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:50 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/385006179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:52.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:52.116 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:52.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:52.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:00:52.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:00:52.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:00:52.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:52.673 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:52 smithi104 ceph-mon[21811]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:00:54.982 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:54.982 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:54.984 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:55.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:55.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:00:55.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:55.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:55 smithi104 ceph-mon[21811]: Deploying daemon mon.smithi135 on smithi135 2024-04-27T01:00:55.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:55 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/1392482373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:57.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:00:57.226 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:00:57.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.317 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:00:59.318 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T00:58:50.728785Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:00:59.318 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2024-04-27T01:00:59.426 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 podman[24023]: 2024-04-27 01:00:59.41033704 +0000 UTC m=+0.437701352 container start 1dbdd689cbb6508229f85737668de1c5ee0e288856a3e9d23a6f220b92946858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, version=8, ceph=True, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, architecture=x86_64) 2024-04-27T01:00:59.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:00:59 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3274370461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pidfile_write: ignore empty --pid-file 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: load: jerasure load: lrc 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: RocksDB version: 6.15.5 2024-04-27T01:00:59.818 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Compile date Apr 18 2022 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: DB SUMMARY 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: DB Session ID: XMT1P5CDK8QY2QUYT8G2 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi135/store.db dir, Total Num: 0, files: 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi135/store.db: 000004.log size: 511 ; 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.env: 0x5571df4c6860 2024-04-27T01:00:59.819 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.fs: Posix File System 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.info_log: 0x5571e01f9fa0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.statistics: (nil) 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.use_fsync: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:00:59.820 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.db_log_dir: 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi135/store.db 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.write_buffer_manager: 0x5571e02ea2d0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:00:59.821 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.unordered_write: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.row_cache: None 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.wal_filter: None 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:00:59.822 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.preserve_deletes: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:00:59.823 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:00:59.825 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_open_files: -1 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Compression algorithms supported: 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kZSTD supported: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:00:59.826 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000003 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.merge_operator: 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_filter: None 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5571e01c5d00) 2024-04-27T01:00:59.827 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_type: 0 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_index_type: 0 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_shortening: 1 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: hash_index_allow_collision: 1 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: checksum: 1 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: no_block_cache: 0 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache: 0x5571e0230170 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_options: 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: capacity : 536870912 2024-04-27T01:00:59.828 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: num_shard_bits : 4 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: strict_capacity_limit : 0 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_compressed: (nil) 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: persistent_cache: (nil) 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size: 4096 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size_deviation: 10 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_restart_interval: 16 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_block_restart_interval: 1 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: metadata_block_size: 4096 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: partition_filters: 0 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: use_delta_encoding: 1 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: whole_key_filtering: 1 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: verify_compression: 0 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:00:59.829 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: format_version: 4 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_index_compression: 1 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_align: 0 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression: NoCompression 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.num_levels: 7 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:00:59.830 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:00:59.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.arena_block_size: 4194304 2024-04-27T01:00:59.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:00:59.833 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.ttl: 2592000 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:00:59.834 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179659436131, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179659436439, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714179659, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a01204b4-f2a0-45b9-b505-deebd4aa034c", "db_session_id": "XMT1P5CDK8QY2QUYT8G2"}} 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179659436741, "job": 1, "event": "recovery_finished"} 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi135/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5571e0216a80 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: DB pointer 0x5571e0226000 2024-04-27T01:00:59.835 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135 does not exist in monmap, will attempt to join an existing cluster 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** DB Stats ** 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:00:59.836 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:00:59.837 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:00:59.838 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: using public_addr v2:172.21.15.135:0/0 -> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: starting mon.smithi135 rank -1 at public addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] at bind addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi135 fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(???) e0 preinit fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:00:59.839 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).mds e1 new map 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).mds e1 print_map 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: e1 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: legacy client fscid: -1 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: No filesystems configured 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-27T01:00:59.840 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon crash.smithi104 on smithi104 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2400900315' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2400900315' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.841 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon grafana.smithi104 on smithi104 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Saving service mon spec with placement smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135;count:2 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/1581502963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3561745819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3726879282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.842 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/146576176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14160 172.21.15.104:0/2549818699' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mgrmap e14: smithi104.dpemzn(active, since 48s) 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/2179445630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3065238310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Activating manager daemon smithi104.dpemzn 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mgrmap e15: smithi104.dpemzn(active, starting, since 0.0524888s) 2024-04-27T01:00:59.843 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mgrmap e16: smithi104.dpemzn(active, since 1.0633s) 2024-04-27T01:00:59.844 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/2536111955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: [27/Apr/2024:01:00:42] ENGINE Bus STARTING 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: [27/Apr/2024:01:00:42] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: [27/Apr/2024:01:00:42] ENGINE Bus STARTED 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mgrmap e17: smithi104.dpemzn(active, since 2s) 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/910644364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.845 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon crash.smithi135 on smithi135 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/385006179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:00:59.846 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: Deploying daemon mon.smithi135 on smithi135 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/1392482373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3274370461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 ceph-mon[24080]: mon.smithi135@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 bash[24023]: 1dbdd689cbb6508229f85737668de1c5ee0e288856a3e9d23a6f220b92946858 2024-04-27T01:00:59.847 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:00:59 smithi135 systemd[1]: Started Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:01:01.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-27T01:01:01.216 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:01:04.763 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: mon.smithi104 calling monitor election 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: mon.smithi135 calling monitor election 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:01:04.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: monmap e2: 2 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: fsmap 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: mgrmap e17: smithi104.dpemzn(active, since 23s) 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: overall HEALTH_OK 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:01:04.765 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:04.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:01:04.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: mon.smithi104 calling monitor election 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: mon.smithi135 calling monitor election 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: monmap e2: 2 mons at {smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],smithi135=[v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0]} 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: fsmap 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: osdmap e5: 0 total, 0 up, 0 in 2024-04-27T01:01:04.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: mgrmap e17: smithi104.dpemzn(active, since 23s) 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: overall HEALTH_OK 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/954150460' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:01:04.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:05.872 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:05 smithi135 ceph-mon[24080]: Deploying daemon node-exporter.smithi135 on smithi135 2024-04-27T01:01:05.872 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:05 smithi135 ceph-mon[24080]: mgrmap e18: smithi104.dpemzn(active, since 23s), standbys: smithi135.nwoumo 2024-04-27T01:01:05.872 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:05 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:01:05.872 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:05 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:05.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:05 smithi104 ceph-mon[21811]: Deploying daemon node-exporter.smithi135 on smithi135 2024-04-27T01:01:05.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:05 smithi104 ceph-mon[21811]: mgrmap e18: smithi104.dpemzn(active, since 23s), standbys: smithi135.nwoumo 2024-04-27T01:01:05.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:05 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:01:05.922 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:05 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:01:06.026 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-04-27T01:01:06.026 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":2,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T01:00:59.463740Z","created":"2024-04-27T00:58:50.728785Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi135","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:3300","nonce":0},{"type":"v1","addr":"172.21.15.135:6789","nonce":0}]},"addr":"172.21.15.135:6789/0","public_addr":"172.21.15.135:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-27T01:01:06.027 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 2 2024-04-27T01:01:07.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:06 smithi135 ceph-mon[24080]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:07.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:06 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3435136196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:01:07.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:07.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:06 smithi104 ceph-mon[21811]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:07.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:06 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3435136196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:01:07.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:07.400 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-27T01:01:07.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-27T01:01:09.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:08 smithi135 ceph-mon[24080]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:09.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:09.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:08 smithi104 ceph-mon[21811]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:09.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:09.277 INFO:teuthology.orchestra.run.smithi104.stdout:# minimal ceph.conf for 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:01:09.278 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-04-27T01:01:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:01:09.278 INFO:teuthology.orchestra.run.smithi104.stdout: mon_host = [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2024-04-27T01:01:10.013 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-27T01:01:10.013 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T01:01:10.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:01:10.038 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T01:01:10.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:01:10.101 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T01:01:10.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:01:10.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:09 smithi135 ceph-mon[24080]: Deploying daemon prometheus.smithi104 on smithi104 2024-04-27T01:01:10.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:09 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3880502374' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:10.133 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T01:01:10.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:01:10.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:09 smithi104 ceph-mon[21811]: Deploying daemon prometheus.smithi104 on smithi104 2024-04-27T01:01:10.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:09 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3880502374' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:10.200 INFO:tasks.cephadm:Deploying OSDs... 2024-04-27T01:01:10.201 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T01:01:10.201 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:01:10.215 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:01:10.216 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-04-27T01:01:10.270 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:01:10.270 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:10.270 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-27 01:00:30.157579510 +0000 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-27 00:57:52.087880602 +0000 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-27 00:57:52.087880602 +0000 2024-04-27T01:01:10.271 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-27 00:57:52.087880602 +0000 2024-04-27T01:01:10.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:01:10.332 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-27T01:01:10.332 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-27T01:01:10.332 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000217149 s, 2.4 MB/s 2024-04-27T01:01:10.333 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:01:10.388 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-27 01:00:30.158579499 +0000 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-27 00:57:52.463876197 +0000 2024-04-27T01:01:10.444 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-27 00:57:52.463876197 +0000 2024-04-27T01:01:10.445 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-27 00:57:52.463876197 +0000 2024-04-27T01:01:10.445 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:01:10.507 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-27T01:01:10.507 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-27T01:01:10.507 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000198222 s, 2.6 MB/s 2024-04-27T01:01:10.509 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:01:10.564 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-04-27T01:01:10.618 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-27 01:00:30.158579499 +0000 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-27 00:57:52.794872319 +0000 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-27 00:57:52.794872319 +0000 2024-04-27T01:01:10.619 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-27 00:57:52.794872319 +0000 2024-04-27T01:01:10.619 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:01:10.681 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-27T01:01:10.681 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-27T01:01:10.681 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000192482 s, 2.7 MB/s 2024-04-27T01:01:10.682 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:01:10.737 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-27 01:00:30.158579499 +0000 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-27 00:57:53.136868313 +0000 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-27 00:57:53.136868313 +0000 2024-04-27T01:01:10.793 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-27 00:57:53.136868313 +0000 2024-04-27T01:01:10.794 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:01:10.854 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-27T01:01:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-27T01:01:10.855 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000205561 s, 2.5 MB/s 2024-04-27T01:01:10.856 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:01:10.912 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-04-27T01:01:10.912 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:01:10.930 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:01:10.931 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-04-27T01:01:10.987 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-04-27 01:01:05.044585581 +0000 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-04-27 00:57:52.087360447 +0000 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-04-27 00:57:52.087360447 +0000 2024-04-27T01:01:10.988 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-04-27 00:57:52.087360447 +0000 2024-04-27T01:01:10.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:01:11.053 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-04-27T01:01:11.054 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-04-27T01:01:11.054 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000279276 s, 1.8 MB/s 2024-04-27T01:01:11.055 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:01:11.114 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-04-27 01:01:05.045585560 +0000 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-04-27 00:57:52.436351815 +0000 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-04-27 00:57:52.436351815 +0000 2024-04-27T01:01:11.171 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-04-27 00:57:52.436351815 +0000 2024-04-27T01:01:11.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:01:11.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:10 smithi104 ceph-mon[21811]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:11.234 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:10 smithi135 ceph-mon[24080]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:11.237 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-04-27T01:01:11.237 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-04-27T01:01:11.237 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000249802 s, 2.0 MB/s 2024-04-27T01:01:11.238 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:01:11.296 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-04-27T01:01:11.353 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-04-27 01:01:05.045585560 +0000 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-04-27 00:57:52.780343307 +0000 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-04-27 00:57:52.780343307 +0000 2024-04-27T01:01:11.354 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-04-27 00:57:52.780343307 +0000 2024-04-27T01:01:11.354 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:01:11.417 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-04-27T01:01:11.418 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-04-27T01:01:11.418 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000244638 s, 2.1 MB/s 2024-04-27T01:01:11.421 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:01:11.476 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-04-27T01:01:11.532 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:01:11.532 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:01:11.532 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-27T01:01:11.532 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:01:11.532 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:01:11.533 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-04-27 01:01:05.045585560 +0000 2024-04-27T01:01:11.533 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-04-27 00:57:53.127334724 +0000 2024-04-27T01:01:11.533 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-04-27 00:57:53.127334724 +0000 2024-04-27T01:01:11.533 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-04-27 00:57:53.127334724 +0000 2024-04-27T01:01:11.533 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:01:11.596 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-04-27T01:01:11.596 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-04-27T01:01:11.596 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000226355 s, 2.3 MB/s 2024-04-27T01:01:11.597 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:01:11.653 INFO:tasks.cephadm:Deploying osd.0 on smithi104 with /dev/vg_nvme/lv_4... 2024-04-27T01:01:11.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T01:01:12.248 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:12 smithi104 ceph-mon[21811]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:12.248 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:12.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:12 smithi135 ceph-mon[24080]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:12.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:14.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:14 smithi104 ceph-mon[21811]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:14.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:14 smithi135 ceph-mon[24080]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132433 s, 792 MB/s 2024-04-27T01:01:14.770 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-27T01:01:16.495 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-04-27T01:01:17.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:16 smithi135 ceph-mon[24080]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:17.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:16 smithi104 ceph-mon[21811]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:19.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:18 smithi135 ceph-mon[24080]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:19.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:18 smithi104 ceph-mon[21811]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:20.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:01:20.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:01:20.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:20.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:01:20.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:01:20.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:21.137 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:20 smithi104 ceph-mon[21811]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:21.137 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:20 smithi104 ceph-mon[21811]: from='client.14236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:01:21.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:20 smithi135 ceph-mon[24080]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:21.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:20 smithi135 ceph-mon[24080]: from='client.14236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:01:22.907 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:22 smithi104 ceph-mon[21811]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:23.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:22 smithi135 ceph-mon[24080]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:24.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:24 smithi135 ceph-mon[24080]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:24.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:24 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:24.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:24 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:01:24.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:24 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:24.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:24 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:01:24.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:24 smithi104 ceph-mon[21811]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:24.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:24 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:24.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:24 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:01:24.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:24 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:24.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:24 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:01:25.215 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3451269767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff8ac495-617b-45e7-b25d-4d2ca1e47d00"}]: dispatch 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3451269767' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff8ac495-617b-45e7-b25d-4d2ca1e47d00"}]': finished 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4290565870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:25.216 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:25 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:01:25.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3451269767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff8ac495-617b-45e7-b25d-4d2ca1e47d00"}]: dispatch 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3451269767' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff8ac495-617b-45e7-b25d-4d2ca1e47d00"}]': finished 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4290565870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:25.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:25 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:01:26.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:26 smithi104 ceph-mon[21811]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:26.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:26 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:01:26.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:26.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:26.523 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:26 smithi135 ceph-mon[24080]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:26.523 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:26 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:01:26.523 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:26.523 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: Reconfiguring mon.smithi104 (unknown last config time)... 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:28.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:28 smithi135 ceph-mon[24080]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-27T01:01:28.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: Reconfiguring mon.smithi104 (unknown last config time)... 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:28.672 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:28 smithi104 ceph-mon[21811]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-27T01:01:31.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:31.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:31.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: Reconfiguring mgr.smithi104.dpemzn (unknown last config time)... 2024-04-27T01:01:31.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:01:31.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:01:31.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:31.122 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:30 smithi135 ceph-mon[24080]: Reconfiguring daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:01:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: Reconfiguring mgr.smithi104.dpemzn (unknown last config time)... 2024-04-27T01:01:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:01:31.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:01:31.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:31.173 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:30 smithi104 ceph-mon[21811]: Reconfiguring daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:01:32.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:01:32.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:32.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:01:32.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:32.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:32 smithi104 ceph-mon[21811]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:32.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:32 smithi104 ceph-mon[21811]: Deploying daemon osd.0 on smithi104 2024-04-27T01:01:32.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:32 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:33.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:32 smithi135 ceph-mon[24080]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:33.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:32 smithi135 ceph-mon[24080]: Deploying daemon osd.0 on smithi104 2024-04-27T01:01:33.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:32 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:33.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:33 smithi104 ceph-mon[21811]: Reconfiguring alertmanager.smithi104 (dependencies changed)... 2024-04-27T01:01:33.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:33 smithi104 ceph-mon[21811]: Reconfiguring daemon alertmanager.smithi104 on smithi104 2024-04-27T01:01:34.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:33 smithi135 ceph-mon[24080]: Reconfiguring alertmanager.smithi104 (dependencies changed)... 2024-04-27T01:01:34.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:33 smithi135 ceph-mon[24080]: Reconfiguring daemon alertmanager.smithi104 on smithi104 2024-04-27T01:01:34.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:34 smithi104 ceph-mon[21811]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:35.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:34 smithi135 ceph-mon[24080]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:37.157 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:36 smithi135 ceph-mon[24080]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:37.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:36 smithi104 ceph-mon[21811]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:39.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:38 smithi135 ceph-mon[24080]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:39.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:39.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:38 smithi104 ceph-mon[21811]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:39.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:41.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:40 smithi104 ceph-mon[21811]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:41.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:40 smithi135 ceph-mon[24080]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:42.048 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:41 smithi104 ceph-mon[21811]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:01:42.048 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:01:42.048 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:01:42.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:41 smithi135 ceph-mon[24080]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:01:42.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:01:42.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:01:43.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:42 smithi104 ceph-mon[21811]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:43.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:42 smithi104 ceph-mon[21811]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:01:43.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:42 smithi104 ceph-mon[21811]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T01:01:43.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:43.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:42 smithi104 ceph-mon[21811]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:01:43.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:42 smithi135 ceph-mon[24080]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:43.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:42 smithi135 ceph-mon[24080]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:01:43.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:42 smithi135 ceph-mon[24080]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T01:01:43.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:43.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:42 smithi135 ceph-mon[24080]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:01:44.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: osdmap e8: 1 total, 0 up, 1 in 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:01:44.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:44.998 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:01:44.998 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:01:44.998 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:44.998 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-27T01:01:44.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-27T01:01:44.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:44.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446] boot 2024-04-27T01:01:44.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T01:01:44.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:45.029 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 0 on host 'smithi104' 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: osd.0 [v2:172.21.15.104:6802/2841042446,v1:172.21.15.104:6803/2841042446] boot 2024-04-27T01:01:45.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T01:01:45.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:01:46.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:45 smithi104 ceph-mon[21811]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T01:01:46.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:46.323 DEBUG:teuthology.orchestra.run.smithi104:osd.0> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.0.service 2024-04-27T01:01:46.326 INFO:tasks.cephadm:Deploying osd.1 on smithi104 with /dev/vg_nvme/lv_3... 2024-04-27T01:01:46.326 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T01:01:46.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:45 smithi135 ceph-mon[24080]: osdmap e10: 1 total, 1 up, 1 in 2024-04-27T01:01:46.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:45 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:46.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:46 smithi104 ceph-mon[21811]: pgmap v30: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:46.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:46 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:47.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:46 smithi135 ceph-mon[24080]: pgmap v30: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:47.372 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:46 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:47 smithi104 ceph-mon[21811]: Reconfiguring grafana.smithi104 (dependencies changed)... 2024-04-27T01:01:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:47 smithi104 ceph-mon[21811]: Reconfiguring daemon grafana.smithi104 on smithi104 2024-04-27T01:01:48.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:47 smithi135 ceph-mon[24080]: Reconfiguring grafana.smithi104 (dependencies changed)... 2024-04-27T01:01:48.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:47 smithi135 ceph-mon[24080]: Reconfiguring daemon grafana.smithi104 on smithi104 2024-04-27T01:01:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T01:01:48.514 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T01:01:48.515 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-27T01:01:48.515 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-27T01:01:48.515 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140631 s, 746 MB/s 2024-04-27T01:01:48.515 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 2024-04-27T01:01:48.515 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-27T01:01:49.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:48 smithi104 ceph-mon[21811]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:49.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:48 smithi135 ceph-mon[24080]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:50.919 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:50 smithi104 ceph-mon[21811]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:51.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:50 smithi135 ceph-mon[24080]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:52.023 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-04-27T01:01:53.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:52 smithi104 ceph-mon[21811]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:53.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:52 smithi135 ceph-mon[24080]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:55.133 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:54 smithi104 ceph-mon[21811]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:55.133 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:54 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:01:55.133 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:54 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:01:55.133 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:54 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:55.314 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:54 smithi135 ceph-mon[24080]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:55.314 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:54 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:01:55.314 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:54 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:01:55.315 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:54 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:56.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:55 smithi104 ceph-mon[21811]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:01:56.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:56.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:01:56.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:56.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:55 smithi135 ceph-mon[24080]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:01:56.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:56.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:01:56.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:57.182 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:57.182 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: Reconfiguring crash.smithi135 (monmap changed)... 2024-04-27T01:01:57.182 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: Reconfiguring daemon crash.smithi135 on smithi135 2024-04-27T01:01:57.182 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:57.183 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:01:57.183 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:01:57.183 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:57.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:57.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: Reconfiguring crash.smithi135 (monmap changed)... 2024-04-27T01:01:57.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: Reconfiguring daemon crash.smithi135 on smithi135 2024-04-27T01:01:57.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:57.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:01:57.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:01:57.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: Reconfiguring mgr.smithi135.nwoumo (monmap changed)... 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: Reconfiguring daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/805559321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e289d866-5a32-4b41-abef-01470666e8a2"}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/805559321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e289d866-5a32-4b41-abef-01470666e8a2"}]': finished 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:58.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1468038458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:01:58.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: Reconfiguring mgr.smithi135.nwoumo (monmap changed)... 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: Reconfiguring daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/805559321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e289d866-5a32-4b41-abef-01470666e8a2"}]: dispatch 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/805559321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e289d866-5a32-4b41-abef-01470666e8a2"}]': finished 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:01:58.422 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1468038458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:01:59.084 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:58 smithi135 ceph-mon[24080]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:59.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:58 smithi135 ceph-mon[24080]: Reconfiguring mon.smithi135 (monmap changed)... 2024-04-27T01:01:59.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:01:58 smithi135 ceph-mon[24080]: Reconfiguring daemon mon.smithi135 on smithi135 2024-04-27T01:01:59.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:58 smithi104 ceph-mon[21811]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:01:59.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:58 smithi104 ceph-mon[21811]: Reconfiguring mon.smithi135 (monmap changed)... 2024-04-27T01:01:59.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:01:58 smithi104 ceph-mon[21811]: Reconfiguring daemon mon.smithi135 on smithi135 2024-04-27T01:02:00.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:00 smithi104 ceph-mon[21811]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:00.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:00.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:02:00.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:00 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:02:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:00 smithi135 ceph-mon[24080]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:02:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:00 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:02:02.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:02 smithi104 ceph-mon[21811]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:02.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:02 smithi135 ceph-mon[24080]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:04.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:04 smithi104 ceph-mon[21811]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:04.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:02:04.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:04.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:04 smithi104 ceph-mon[21811]: Deploying daemon osd.1 on smithi104 2024-04-27T01:02:04.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:04 smithi135 ceph-mon[24080]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:04.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:02:04.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:04.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:04 smithi135 ceph-mon[24080]: Deploying daemon osd.1 on smithi104 2024-04-27T01:02:07.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:06 smithi104 ceph-mon[21811]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:07.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:06 smithi135 ceph-mon[24080]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:09.007 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:08 smithi104 ceph-mon[21811]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:09.007 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:09.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:08 smithi135 ceph-mon[24080]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:09.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:11.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:10 smithi104 ceph-mon[21811]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:11.166 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 1 on host 'smithi104' 2024-04-27T01:02:11.234 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:10 smithi135 ceph-mon[24080]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:11.769 DEBUG:teuthology.orchestra.run.smithi104:osd.1> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.1.service 2024-04-27T01:02:11.771 INFO:tasks.cephadm:Deploying osd.2 on smithi104 with /dev/vg_nvme/lv_2... 2024-04-27T01:02:11.772 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.104:9093"}]: dispatch 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.104:9093"}]: dispatch 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:12.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:12 smithi104 ceph-mon[21811]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:02:12.481 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:12.481 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.104:9093"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.104:9093"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.104:3000"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.104:9095"}]: dispatch 2024-04-27T01:02:12.482 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:12.483 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:12.483 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:12.483 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:12.483 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:12 smithi135 ceph-mon[24080]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:02:13.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:13 smithi135 ceph-mon[24080]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:02:13.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:13 smithi135 ceph-mon[24080]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T01:02:13.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:13 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:13.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:13 smithi135 ceph-mon[24080]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:02:13.371 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:13 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:13.390 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-mon[21811]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:02:13.390 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-mon[21811]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T01:02:13.390 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:13.391 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-mon[21811]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:02:13.391 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:13.391 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[35086]: 2024-04-27T01:02:13.211+0000 7f2b3dc8c700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T01:02:13.391 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[35086]: 2024-04-27T01:02:13.214+0000 7f2b3a627700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:02:14.386 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T01:02:14.387 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T01:02:14.387 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-27T01:02:14.387 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-27T01:02:14.387 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121726 s, 861 MB/s 2024-04-27T01:02:14.387 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-27T01:02:14.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:14 smithi104 ceph-mon[21811]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:14.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:14 smithi104 ceph-mon[21811]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:02:14.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:14 smithi104 ceph-mon[21811]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T01:02:14.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:14.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:14.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:14 smithi135 ceph-mon[24080]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:14.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:14 smithi135 ceph-mon[24080]: from='osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:02:14.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:14 smithi135 ceph-mon[24080]: osdmap e13: 2 total, 1 up, 2 in 2024-04-27T01:02:14.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:14.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:15.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:15 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:02:15.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:15 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:02:15.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:15 smithi104 ceph-mon[21811]: osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489] boot 2024-04-27T01:02:15.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:15 smithi104 ceph-mon[21811]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T01:02:15.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:15 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:15.364 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-04-27T01:02:15.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:15 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:02:15.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:15 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:02:15.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:15 smithi135 ceph-mon[24080]: osd.1 [v2:172.21.15.104:6810/242749489,v1:172.21.15.104:6811/242749489] boot 2024-04-27T01:02:15.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:15 smithi135 ceph-mon[24080]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T01:02:15.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:15 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:02:16.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:16 smithi104 ceph-mon[21811]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:16.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:16 smithi104 ceph-mon[21811]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T01:02:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:16 smithi135 ceph-mon[24080]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:02:16.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:16 smithi135 ceph-mon[24080]: osdmap e15: 2 total, 2 up, 2 in 2024-04-27T01:02:18.331 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:18 smithi104 ceph-mon[21811]: pgmap v51: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:18.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:18 smithi135 ceph-mon[24080]: pgmap v51: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:19.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:19 smithi104 ceph-mon[21811]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:02:19.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:02:19.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:02:19.421 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:19.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:19 smithi135 ceph-mon[24080]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:02:19.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:02:19.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:02:19.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:20.935 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:20 smithi104 ceph-mon[21811]: pgmap v52: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:20.935 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:20 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:21.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:20 smithi135 ceph-mon[24080]: pgmap v52: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:21.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:20 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:22.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:21 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3788902863' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91841358-2470-4ecb-a2a3-3d4d56137cdc"}]: dispatch 2024-04-27T01:02:22.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:21 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3788902863' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91841358-2470-4ecb-a2a3-3d4d56137cdc"}]': finished 2024-04-27T01:02:22.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:21 smithi135 ceph-mon[24080]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T01:02:22.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:21 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:22.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:21 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3788902863' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91841358-2470-4ecb-a2a3-3d4d56137cdc"}]: dispatch 2024-04-27T01:02:22.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:21 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3788902863' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91841358-2470-4ecb-a2a3-3d4d56137cdc"}]': finished 2024-04-27T01:02:22.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:21 smithi104 ceph-mon[21811]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T01:02:22.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:21 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:22.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:22 smithi135 ceph-mon[24080]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:22.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:22 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4013300045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:02:23.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:22 smithi104 ceph-mon[21811]: pgmap v53: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:23.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:22 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4013300045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:02:24.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:24 smithi104 ceph-mon[21811]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:25.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:24 smithi135 ceph-mon[24080]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:27.143 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:27.144 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:27.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:27.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:27.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:27.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:27 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi104 2024-04-27T01:02:27.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:27 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi104 to 6432M 2024-04-27T01:02:28.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:27 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi104 2024-04-27T01:02:28.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:27 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi104 to 6432M 2024-04-27T01:02:29.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:28 smithi104 ceph-mon[21811]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:29.046 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:29.046 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:02:29.046 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:28 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:29.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:28 smithi135 ceph-mon[24080]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:29.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:29.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:02:29.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:28 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:30.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:29 smithi104 ceph-mon[21811]: Deploying daemon osd.2 on smithi104 2024-04-27T01:02:30.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:29 smithi135 ceph-mon[24080]: Deploying daemon osd.2 on smithi104 2024-04-27T01:02:30.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:30 smithi104 ceph-mon[21811]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:31.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:30 smithi135 ceph-mon[24080]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:33.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:32 smithi104 ceph-mon[21811]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:33.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:32 smithi135 ceph-mon[24080]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:35.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:34 smithi135 ceph-mon[24080]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:35.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:35.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:34 smithi104 ceph-mon[21811]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:35.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:36.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:36 smithi104 ceph-mon[21811]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:37.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:36 smithi135 ceph-mon[24080]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:38.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:38 smithi135 ceph-mon[24080]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:38.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:38.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:38 smithi104 ceph-mon[21811]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:38.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:40.552 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:40.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:40.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:40.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:40.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:40.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:40.739 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:40.740 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:40.740 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:40.740 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:40.740 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:40.740 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:41.297 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 2 on host 'smithi104' 2024-04-27T01:02:41.612 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:41 smithi104 ceph-mon[21811]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:02:41.612 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:41.612 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:02:41.612 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:02:41.612 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:41.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:41 smithi135 ceph-mon[24080]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:02:41.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:41.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:02:41.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:02:41.871 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:41.911 DEBUG:teuthology.orchestra.run.smithi104:osd.2> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.2.service 2024-04-27T01:02:41.914 INFO:tasks.cephadm:Deploying osd.3 on smithi104 with /dev/vg_nvme/lv_1... 2024-04-27T01:02:41.914 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T01:02:42.536 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-mon[21811]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:42.559 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-mon[21811]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:02:42.559 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-mon[21811]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T01:02:42.559 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:42.559 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-mon[21811]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:02:42.560 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[38959]: 2024-04-27T01:02:42.416+0000 7f3b6c20e700 -1 osd.2 0 waiting for initial osdmap 2024-04-27T01:02:42.560 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:42 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[38959]: 2024-04-27T01:02:42.420+0000 7f3b673a6700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:02:42.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:42 smithi135 ceph-mon[24080]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:42.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:42 smithi135 ceph-mon[24080]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:02:42.831 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:42 smithi135 ceph-mon[24080]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T01:02:42.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:42.832 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:42 smithi135 ceph-mon[24080]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:02:43.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:43 smithi104 ceph-mon[21811]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:02:43.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:43 smithi104 ceph-mon[21811]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T01:02:43.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:43.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:43.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:43 smithi135 ceph-mon[24080]: from='osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:02:43.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:43 smithi135 ceph-mon[24080]: osdmap e18: 3 total, 2 up, 3 in 2024-04-27T01:02:43.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:43.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0186205 s, 563 MB/s 2024-04-27T01:02:44.445 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-27T01:02:44.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:02:44.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:02:44.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:44.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348] boot 2024-04-27T01:02:44.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T01:02:44.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: osd.2 [v2:172.21.15.104:6818/1143353348,v1:172.21.15.104:6819/1143353348] boot 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T01:02:44.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:02:45.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-04-27T01:02:45.725 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:45 smithi104 ceph-mon[21811]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T01:02:45.726 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-27T01:02:45.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:45 smithi135 ceph-mon[24080]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T01:02:45.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:45 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-27T01:02:47.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:46 smithi104 ceph-mon[21811]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:47.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:46 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-27T01:02:47.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:46 smithi104 ceph-mon[21811]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T01:02:47.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:46 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:02:47.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:46 smithi135 ceph-mon[24080]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:47.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:46 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-27T01:02:47.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:46 smithi135 ceph-mon[24080]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T01:02:47.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:46 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:02:47.640 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40357]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:02:47.640 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40357]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:47.640 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40357]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:47.898 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40436]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:02:47.898 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40436]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:47.898 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40436]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:48.170 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40515]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:02:48.170 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40515]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:48.171 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:47 smithi104 sudo[40515]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:48.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:47 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:48.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:47 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:48.464 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40523]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:02:48.464 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40523]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:48.465 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40523]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:48.714 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40591]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:02:48.715 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40591]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:48.715 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40591]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:48.968 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40603]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:02:48.968 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40603]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:48.968 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40603]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:49.228 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40617]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:02:49.229 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40617]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:49.229 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:48 smithi104 sudo[40617]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 ceph-mon[24080]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 ceph-mon[24080]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 ceph-mon[24080]: osdmap e23: 3 total, 3 up, 3 in 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 ceph-mon[24080]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26403]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26403]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:49.487 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26403]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:49.592 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 sudo[40624]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:02:49.592 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 sudo[40624]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:49.592 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 sudo[40624]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:49.592 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 ceph-mon[21811]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:49.592 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 ceph-mon[21811]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:02:49.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 ceph-mon[21811]: osdmap e23: 3 total, 3 up, 3 in 2024-04-27T01:02:49.593 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:49 smithi104 ceph-mon[21811]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:02:49.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26406]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:02:49.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26406]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-27T01:02:49.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:49 smithi135 sudo[26406]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:02:50.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:02:50.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:50.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:02:50.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:02:50.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: mgrmap e19: smithi104.dpemzn(active, since 2m), standbys: smithi135.nwoumo 2024-04-27T01:02:50.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:50.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:50.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: mgrmap e19: smithi104.dpemzn(active, since 2m), standbys: smithi135.nwoumo 2024-04-27T01:02:50.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:51.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:51 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2563567774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ae4ed45-850e-46b9-8648-2c48f2d38316"}]: dispatch 2024-04-27T01:02:51.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:51 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2563567774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ae4ed45-850e-46b9-8648-2c48f2d38316"}]': finished 2024-04-27T01:02:51.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:51 smithi135 ceph-mon[24080]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T01:02:51.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:51 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:02:51.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:51 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2262989686' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:02:51.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:51 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2563567774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5ae4ed45-850e-46b9-8648-2c48f2d38316"}]: dispatch 2024-04-27T01:02:51.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:51 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2563567774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5ae4ed45-850e-46b9-8648-2c48f2d38316"}]': finished 2024-04-27T01:02:51.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:51 smithi104 ceph-mon[21811]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T01:02:51.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:51 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:02:51.671 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:51 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2262989686' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:02:52.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:52 smithi135 ceph-mon[24080]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:52.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:52 smithi104 ceph-mon[21811]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:54.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:54 smithi104 ceph-mon[21811]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:54.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:54 smithi135 ceph-mon[24080]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi104 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi104 to 4288M 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:57.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:57.121 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:56 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:57.127 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi104 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi104 to 4288M 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:57.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:56 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:02:59.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:58 smithi135 ceph-mon[24080]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:59.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:02:59.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:59.126 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:58 smithi104 ceph-mon[21811]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:02:59.127 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:02:59.127 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:02:59.998 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:02:59 smithi104 ceph-mon[21811]: Deploying daemon osd.3 on smithi104 2024-04-27T01:03:00.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:02:59 smithi135 ceph-mon[24080]: Deploying daemon osd.3 on smithi104 2024-04-27T01:03:00.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:00 smithi104 ceph-mon[21811]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:01.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:00 smithi135 ceph-mon[24080]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:03.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:02 smithi104 ceph-mon[21811]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:03.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:02 smithi135 ceph-mon[24080]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:04.807 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:04 smithi104 ceph-mon[21811]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:04.807 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:04.807 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:04 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:04.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:04 smithi135 ceph-mon[24080]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:04.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:04.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:04 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:06.825 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:06 smithi104 ceph-mon[21811]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:06.825 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:06.825 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:06.825 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:06.825 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:06.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:06 smithi135 ceph-mon[24080]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:06.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:06.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:06.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:06.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:08.273 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 3 on host 'smithi104' 2024-04-27T01:03:08.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:08 smithi135 ceph-mon[24080]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:08.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:08.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:08 smithi104 ceph-mon[21811]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:08.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:09.193 DEBUG:teuthology.orchestra.run.smithi104:osd.3> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.3.service 2024-04-27T01:03:09.195 INFO:tasks.cephadm:Deploying osd.4 on smithi135 with /dev/vg_nvme/lv_4... 2024-04-27T01:03:09.196 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T01:03:09.758 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:09 smithi135 ceph-mon[24080]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:03:09.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:09 smithi104 ceph-mon[21811]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:03:10.738 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:10 smithi135 ceph-mon[24080]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:10.738 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:10 smithi135 ceph-mon[24080]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:03:10.738 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:10 smithi135 ceph-mon[24080]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T01:03:10.738 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:10 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:10.738 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:10 smithi135 ceph-mon[24080]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:03:10.855 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-mon[21811]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:10.856 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-mon[21811]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:03:10.856 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-mon[21811]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T01:03:10.856 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:10.856 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-mon[21811]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:03:10.856 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[42823]: 2024-04-27T01:03:10.517+0000 7f5e5b605700 -1 osd.3 0 waiting for initial osdmap 2024-04-27T01:03:10.856 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:03:10 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[42823]: 2024-04-27T01:03:10.521+0000 7f5e54f9a700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110825 s, 946 MB/s 2024-04-27T01:03:11.040 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-04-27T01:03:11.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:11 smithi135 ceph-mon[24080]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:03:11.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:11 smithi135 ceph-mon[24080]: osdmap e26: 4 total, 3 up, 4 in 2024-04-27T01:03:11.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:11 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:11.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:11 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:11.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_4 2024-04-27T01:03:11.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:11 smithi104 ceph-mon[21811]: from='osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-27T01:03:11.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:11 smithi104 ceph-mon[21811]: osdmap e26: 4 total, 3 up, 4 in 2024-04-27T01:03:11.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:11 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:11.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:11 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:12.630 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:03:12.630 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:03:12.630 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:12.631 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438] boot 2024-04-27T01:03:12.631 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T01:03:12.631 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:12.631 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: osd.3 [v2:172.21.15.104:6826/1445164438,v1:172.21.15.104:6827/1445164438] boot 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:03:12.870 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:13.595 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:13 smithi104 ceph-mon[21811]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T01:03:13.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:13 smithi135 ceph-mon[24080]: osdmap e28: 4 total, 4 up, 4 in 2024-04-27T01:03:14.848 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:14 smithi135 ceph-mon[24080]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:14.848 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:14 smithi135 ceph-mon[24080]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:03:14.848 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:03:14.848 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:03:14.848 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:14 smithi104 ceph-mon[21811]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:14 smithi104 ceph-mon[21811]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:03:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:03:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:03:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:16.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi104 to 3216M 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:16.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/4001952688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]: dispatch 2024-04-27T01:03:16.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]: dispatch 2024-04-27T01:03:16.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]': finished 2024-04-27T01:03:16.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T01:03:16.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi104 to 3216M 2024-04-27T01:03:16.658 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/4001952688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]: dispatch 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]: dispatch 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2730a7e-cb82-4f02-bf35-56d2e7ac3544"}]': finished 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T01:03:16.659 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:17.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:17 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/1557528098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:03:17.670 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:17 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/1557528098' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:03:18.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:18 smithi104 ceph-mon[21811]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:18.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:18 smithi135 ceph-mon[24080]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:20.715 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:20 smithi135 ceph-mon[24080]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:20.715 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:20 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:20.715 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:20 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:20.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:20 smithi104 ceph-mon[21811]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:20.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:20 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:20.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:20 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:22.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:22 smithi135 ceph-mon[24080]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:22.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:03:22.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:22.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:22 smithi135 ceph-mon[24080]: Deploying daemon osd.4 on smithi135 2024-04-27T01:03:22.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:22 smithi104 ceph-mon[21811]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:22.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:03:22.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:22.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:22 smithi104 ceph-mon[21811]: Deploying daemon osd.4 on smithi135 2024-04-27T01:03:24.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:24 smithi104 ceph-mon[21811]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:24.801 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:24 smithi135 ceph-mon[24080]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:26.945 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:26 smithi104 ceph-mon[21811]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:26.945 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:26.945 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:26.945 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:26.945 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:27.018 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:26 smithi135 ceph-mon[24080]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:27.018 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:27.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:27.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:27.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:29.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:28 smithi135 ceph-mon[24080]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:29.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:28 smithi104 ceph-mon[21811]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:29.946 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:29 smithi135 ceph-mon[24080]: from='osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:03:29.947 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:29 smithi135 ceph-mon[24080]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:03:29.979 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 4 on host 'smithi135' 2024-04-27T01:03:30.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:29 smithi104 ceph-mon[21811]: from='osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:03:30.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:29 smithi104 ceph-mon[21811]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:03:30.913 DEBUG:teuthology.orchestra.run.smithi135:osd.4> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.4.service 2024-04-27T01:03:30.915 INFO:tasks.cephadm:Deploying osd.5 on smithi135 with /dev/vg_nvme/lv_3... 2024-04-27T01:03:30.915 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:31.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:31.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:32.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:31 smithi135 ceph-mon[24080]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:03:32.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:31 smithi135 ceph-mon[24080]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T01:03:32.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:32.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:32.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:31 smithi104 ceph-mon[21811]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:03:32.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:31 smithi104 ceph-mon[21811]: osdmap e31: 5 total, 4 up, 5 in 2024-04-27T01:03:32.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:32.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:33.064 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:03:33.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:03:33.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:33.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417] boot 2024-04-27T01:03:33.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T01:03:33.065 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:32 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: osd.4 [v2:172.21.15.135:6800/4199538417,v1:172.21.15.135:6801/4199538417] boot 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T01:03:33.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:32 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179163 s, 585 MB/s 2024-04-27T01:03:34.000 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-04-27T01:03:34.000 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:33 smithi135 ceph-mon[24080]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T01:03:34.001 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:33 smithi135 ceph-mon[24080]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T01:03:34.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:33 smithi104 ceph-mon[21811]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T01:03:34.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:33 smithi104 ceph-mon[21811]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T01:03:35.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:34 smithi135 ceph-mon[24080]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:35.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:34 smithi135 ceph-mon[24080]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T01:03:35.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:34 smithi104 ceph-mon[21811]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:35.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:34 smithi104 ceph-mon[21811]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T01:03:35.462 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_3 2024-04-27T01:03:36.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:36.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi135 2024-04-27T01:03:36.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi135 to 15936M 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:36 smithi135 ceph-mon[24080]: osdmap e36: 5 total, 5 up, 5 in 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi135 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi135 to 15936M 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:36.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:36 smithi104 ceph-mon[21811]: osdmap e36: 5 total, 5 up, 5 in 2024-04-27T01:03:38.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:38 smithi135 ceph-mon[24080]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-27T01:03:38.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:38 smithi135 ceph-mon[24080]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:03:38.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:03:38.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:03:38.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:38.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:38 smithi104 ceph-mon[21811]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-27T01:03:38.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:38 smithi104 ceph-mon[21811]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:03:38.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:03:38.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:03:38.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/27660945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]: dispatch 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]: dispatch 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]': finished 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T01:03:40.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:40.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:40 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/530039274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/27660945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]: dispatch 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]: dispatch 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bdbfd71-e400-4c27-800d-a9ab591d3d3f"}]': finished 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:40.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:40 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/530039274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:03:41.321 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:03:41.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:03:42.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:42 smithi104 ceph-mon[21811]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-27T01:03:42.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:03:42.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:42 smithi135 ceph-mon[24080]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-27T01:03:42.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:03:44.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:44 smithi135 ceph-mon[24080]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T01:03:44.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:44 smithi104 ceph-mon[21811]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T01:03:46.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:45 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:03:46.120 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:45 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:46.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:03:46.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:47.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:46 smithi135 ceph-mon[24080]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:47.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:46 smithi135 ceph-mon[24080]: Deploying daemon osd.5 on smithi135 2024-04-27T01:03:47.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:46 smithi104 ceph-mon[21811]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:47.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:46 smithi104 ceph-mon[21811]: Deploying daemon osd.5 on smithi135 2024-04-27T01:03:49.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:48 smithi135 ceph-mon[24080]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:49.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:48 smithi104 ceph-mon[21811]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:51.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:50 smithi104 ceph-mon[21811]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:51.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:51.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:51.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:51.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:50 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:51.303 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:50 smithi135 ceph-mon[24080]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:51.304 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:51.304 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:51.304 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:51.304 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:50 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:53.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:52 smithi104 ceph-mon[21811]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:53.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:52 smithi135 ceph-mon[24080]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:53.549 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 5 on host 'smithi135' 2024-04-27T01:03:54.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:53 smithi135 ceph-mon[24080]: from='osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:03:54.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:53 smithi135 ceph-mon[24080]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:03:54.132 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:53 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:54.133 DEBUG:teuthology.orchestra.run.smithi135:osd.5> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.5.service 2024-04-27T01:03:54.135 INFO:tasks.cephadm:Deploying osd.6 on smithi135 with /dev/vg_nvme/lv_2... 2024-04-27T01:03:54.135 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T01:03:54.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:53 smithi104 ceph-mon[21811]: from='osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:03:54.218 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:53 smithi104 ceph-mon[21811]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:03:54.218 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:53 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:55.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:55.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:03:55.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T01:03:55.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:55.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: from='osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:55.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:55.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:55.370 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[31299]: 2024-04-27T01:03:54.936+0000 7fcae3c14700 -1 osd.5 0 waiting for initial osdmap 2024-04-27T01:03:55.370 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:03:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[31299]: 2024-04-27T01:03:54.943+0000 7fcade5ab700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:03:55.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: from='osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:03:55.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:54 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:56.098 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:03:56.098 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: osdmap e39: 6 total, 5 up, 6 in 2024-04-27T01:03:56.098 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:56.098 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:56.099 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730] boot 2024-04-27T01:03:56.099 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T01:03:56.099 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:55 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:56.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:03:56.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: osdmap e39: 6 total, 5 up, 6 in 2024-04-27T01:03:56.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:56.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:56.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: osd.5 [v2:172.21.15.135:6808/2417530730,v1:172.21.15.135:6809/2417530730] boot 2024-04-27T01:03:56.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T01:03:56.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:55 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:03:57.164 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:56 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:03:57.164 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:56 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:03:57.164 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:56 smithi135 ceph-mon[24080]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:57.164 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:56 smithi135 ceph-mon[24080]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T01:03:57.229 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T01:03:57.230 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T01:03:57.230 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-04-27T01:03:57.230 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-04-27T01:03:57.230 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150257 s, 698 MB/s 2024-04-27T01:03:57.230 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-04-27T01:03:57.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:56 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:03:57.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:56 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:03:57.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:56 smithi104 ceph-mon[21811]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:03:57.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:56 smithi104 ceph-mon[21811]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T01:03:58.119 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_2 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:59.281 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:58 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:03:59.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T01:03:59.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:03:59.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:58 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:00.336 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:59 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi135 2024-04-27T01:04:00.337 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:59 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi135 to 7968M 2024-04-27T01:04:00.337 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:03:59 smithi135 ceph-mon[24080]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T01:04:00.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:59 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi135 2024-04-27T01:04:00.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:59 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi135 to 7968M 2024-04-27T01:04:00.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:03:59 smithi104 ceph-mon[21811]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T01:04:01.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:00 smithi135 ceph-mon[24080]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:01.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:00 smithi135 ceph-mon[24080]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T01:04:01.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:04:01.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:04:01.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:00 smithi104 ceph-mon[21811]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:00 smithi104 ceph-mon[21811]: osdmap e44: 6 total, 6 up, 6 in 2024-04-27T01:04:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:04:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:04:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:02.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:01 smithi135 ceph-mon[24080]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:02.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:01 smithi104 ceph-mon[21811]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/1965363788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]: dispatch 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]: dispatch 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]': finished 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T01:04:03.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:03.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:03.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/1965363788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]: dispatch 2024-04-27T01:04:03.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]: dispatch 2024-04-27T01:04:03.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5518d906-ba93-46aa-a617-42f5fa9f7199"}]': finished 2024-04-27T01:04:03.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T01:04:03.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:04.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:03 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/2493930784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:04:04.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:03 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/2493930784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:04:05.300 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:04 smithi135 ceph-mon[24080]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:05.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:04 smithi104 ceph-mon[21811]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:07.159 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:06 smithi135 ceph-mon[24080]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:07.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:06 smithi104 ceph-mon[21811]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:09.094 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:08 smithi135 ceph-mon[24080]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:09.094 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:04:09.094 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:09.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:08 smithi104 ceph-mon[21811]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:09.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:04:09.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:10.102 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:09 smithi135 ceph-mon[24080]: Deploying daemon osd.6 on smithi135 2024-04-27T01:04:10.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:09 smithi104 ceph-mon[21811]: Deploying daemon osd.6 on smithi135 2024-04-27T01:04:11.231 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:10 smithi135 ceph-mon[24080]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:11.320 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:10 smithi104 ceph-mon[21811]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:13.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:12 smithi135 ceph-mon[24080]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:13.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:12 smithi104 ceph-mon[21811]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:14.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:14 smithi135 ceph-mon[24080]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:14.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:14.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:14.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:14.869 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:14 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:14.919 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:14 smithi104 ceph-mon[21811]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:14.919 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:14.919 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:14.920 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:14 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:17.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:16 smithi135 ceph-mon[24080]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:17.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:16 smithi135 ceph-mon[24080]: from='osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:04:17.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:16 smithi135 ceph-mon[24080]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:04:17.168 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 6 on host 'smithi135' 2024-04-27T01:04:17.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:16 smithi104 ceph-mon[21811]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:17.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:16 smithi104 ceph-mon[21811]: from='osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:04:17.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:16 smithi104 ceph-mon[21811]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:04:17.795 DEBUG:teuthology.orchestra.run.smithi135:osd.6> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.6.service 2024-04-27T01:04:17.797 INFO:tasks.cephadm:Deploying osd.7 on smithi135 with /dev/vg_nvme/lv_1... 2024-04-27T01:04:17.797 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: from='osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:18.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:18.119 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[34286]: 2024-04-27T01:04:17.926+0000 7fb5acb33700 -1 osd.6 0 waiting for initial osdmap 2024-04-27T01:04:18.119 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:04:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[34286]: 2024-04-27T01:04:17.934+0000 7fb5a74ca700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:04:18.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:04:18.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T01:04:18.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:18.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: from='osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:18.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:18.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:17 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: osdmap e47: 7 total, 6 up, 7 in 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:19.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:18 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: osdmap e47: 7 total, 6 up, 7 in 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:19.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:18 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148193 s, 708 MB/s 2024-04-27T01:04:20.036 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199] boot 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:20.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:19 smithi135 ceph-mon[24080]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: osd.6 [v2:172.21.15.135:6816/3609133199,v1:172.21.15.135:6817/3609133199] boot 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:04:20.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:19 smithi104 ceph-mon[21811]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T01:04:20.801 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_1 2024-04-27T01:04:21.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:20 smithi135 ceph-mon[24080]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:21.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:20 smithi135 ceph-mon[24080]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T01:04:21.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:20 smithi104 ceph-mon[21811]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:21.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:20 smithi104 ceph-mon[21811]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T01:04:23.047 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:22 smithi135 ceph-mon[24080]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:23.047 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:22 smithi135 ceph-mon[24080]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T01:04:23.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:22 smithi104 ceph-mon[21811]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:23.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:22 smithi104 ceph-mon[21811]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T01:04:24.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:23 smithi135 ceph-mon[24080]: osdmap e52: 7 total, 7 up, 7 in 2024-04-27T01:04:24.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:23 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:04:24.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:23 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:04:24.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:23 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:24.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:23 smithi104 ceph-mon[21811]: osdmap e52: 7 total, 7 up, 7 in 2024-04-27T01:04:24.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:23 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:04:24.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:23 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:04:24.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:23 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:25.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:24 smithi135 ceph-mon[24080]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:25.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:24 smithi135 ceph-mon[24080]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:25.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:24 smithi104 ceph-mon[21811]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:25.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:24 smithi104 ceph-mon[21811]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi135 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi135 to 5312M 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/4068118831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]: dispatch 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]': finished 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T01:04:26.420 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi135 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi135 to 5312M 2024-04-27T01:04:26.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/4068118831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]: dispatch 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]: dispatch 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60426297-8db7-492e-abf3-6877a2118453"}]': finished 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T01:04:26.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:27.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:27 smithi104 ceph-mon[21811]: from='client.? 172.21.15.135:0/3344976893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:04:27.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:27 smithi135 ceph-mon[24080]: from='client.? 172.21.15.135:0/3344976893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:04:28.419 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:28 smithi104 ceph-mon[21811]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T01:04:28.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:28 smithi135 ceph-mon[24080]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-27T01:04:31.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:30 smithi104 ceph-mon[21811]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-27T01:04:31.190 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:30 smithi135 ceph-mon[24080]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-27T01:04:33.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:32 smithi135 ceph-mon[24080]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-27T01:04:33.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:32 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:04:33.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:32 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:33.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:32 smithi104 ceph-mon[21811]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-27T01:04:33.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:32 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:04:33.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:32 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:34.012 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:33 smithi135 ceph-mon[24080]: Deploying daemon osd.7 on smithi135 2024-04-27T01:04:34.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:33 smithi104 ceph-mon[21811]: Deploying daemon osd.7 on smithi135 2024-04-27T01:04:35.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:34 smithi104 ceph-mon[21811]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T01:04:35.317 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:34 smithi135 ceph-mon[24080]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-27T01:04:37.037 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:36 smithi135 ceph-mon[24080]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:37.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:36 smithi104 ceph-mon[21811]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:38.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:38 smithi135 ceph-mon[24080]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:38.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:38.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:38.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:38.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:38 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:38.669 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:38 smithi104 ceph-mon[21811]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:38.669 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:38.669 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:38.669 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:38.669 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:38 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:41.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:40 smithi135 ceph-mon[24080]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:41.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:40 smithi135 ceph-mon[24080]: from='osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:04:41.138 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:40 smithi135 ceph-mon[24080]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:04:41.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:40 smithi104 ceph-mon[21811]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:41.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:40 smithi104 ceph-mon[21811]: from='osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:04:41.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:40 smithi104 ceph-mon[21811]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:04:41.173 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 7 on host 'smithi135' 2024-04-27T01:04:41.788 DEBUG:teuthology.orchestra.run.smithi135:osd.7> sudo journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.7.service 2024-04-27T01:04:41.791 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-27T01:04:41.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:04:42.077 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:04:42.078 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:42.078 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:42.078 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[37359]: 2024-04-27T01:04:41.925+0000 7fa42ee7a700 -1 osd.7 0 waiting for initial osdmap 2024-04-27T01:04:42.078 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:04:41 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[37359]: 2024-04-27T01:04:41.932+0000 7fa42800e700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:04:42.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:42.104 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:43.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:42 smithi135 ceph-mon[24080]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:43.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:42 smithi135 ceph-mon[24080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:04:43.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:42 smithi135 ceph-mon[24080]: osdmap e55: 8 total, 7 up, 8 in 2024-04-27T01:04:43.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:43.119 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:43.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:42 smithi104 ceph-mon[21811]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:43.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:42 smithi104 ceph-mon[21811]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-04-27T01:04:43.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:42 smithi104 ceph-mon[21811]: osdmap e55: 8 total, 7 up, 8 in 2024-04-27T01:04:43.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:43.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:43.797 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: purged_snaps scrub starts 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: purged_snaps scrub ok 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594] boot 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:44.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:43 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1236573138' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: purged_snaps scrub starts 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: purged_snaps scrub ok 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: osd.7 [v2:172.21.15.135:6824/3828876594,v1:172.21.15.135:6825/3828876594] boot 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:04:44.255 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:43 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1236573138' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T01:04:44.441 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1714179882,"num_in_osds":8,"osd_in_since":1714179866,"num_remapped_pgs":1} 2024-04-27T01:04:44.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:04:44.663 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:04:44.996 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:44.996 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:44.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:04:45.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:45.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:45.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:45.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:04:45.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:04:45.370 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:04:45.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:45 smithi104 ceph-mon[21811]: Detected new or changed devices on smithi135 2024-04-27T01:04:45.995 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:45 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi135 to 3984M 2024-04-27T01:04:45.995 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:45 smithi104 ceph-mon[21811]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T01:04:46.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:45 smithi135 ceph-mon[24080]: Detected new or changed devices on smithi135 2024-04-27T01:04:46.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:45 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi135 to 3984M 2024-04-27T01:04:46.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:45 smithi135 ceph-mon[24080]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T01:04:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:04:46.384 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":59,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","created":"2024-04-27T00:58:55.078206+0000","modified":"2024-04-27T01:04:45.941895+0000","last_up_change":"2024-04-27T01:04:42.923598+0000","last_in_change":"2024-04-27T01:04:26.087299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:02:45.309312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff8ac495-617b-45e7-b25d-4d2ca1e47d00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6803","nonce":2841042446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6805","nonce":2841042446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6809","nonce":2841042446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6807","nonce":2841042446}]},"public_addr":"172.21.15.104:6803/2841042446","cluster_addr":"172.21.15.104:6805/2841042446","heartbeat_back_addr":"172.21.15.104:6809/2841042446","heartbeat_front_addr":"172.21.15.104:6807/2841042446","state":["exists","up"]},{"osd":1,"uuid":"e289d866-5a32-4b41-abef-01470666e8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6811","nonce":242749489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6813","nonce":242749489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6817","nonce":242749489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6815","nonce":242749489}]},"public_addr":"172.21.15.104:6811/242749489","cluster_addr":"172.21.15.104:6813/242749489","heartbeat_back_addr":"172.21.15.104:6817/242749489","heartbeat_front_addr":"172.21.15.104:6815/242749489","state":["exists","up"]},{"osd":2,"uuid":"91841358-2470-4ecb-a2a3-3d4d56137cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6819","nonce":1143353348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6821","nonce":1143353348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6825","nonce":1143353348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6823","nonce":1143353348}]},"public_addr":"172.21.15.104:6819/1143353348","cluster_addr":"172.21.15.104:6821/1143353348","heartbeat_back_addr":"172.21.15.104:6825/1143353348","heartbeat_front_addr":"172.21.15.104:6823/1143353348","state":["exists","up"]},{"osd":3,"uuid":"5ae4ed45-850e-46b9-8648-2c48f2d38316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6827","nonce":1445164438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6829","nonce":1445164438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6833","nonce":1445164438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6831","nonce":1445164438}]},"public_addr":"172.21.15.104:6827/1445164438","cluster_addr":"172.21.15.104:6829/1445164438","heartbeat_back_addr":"172.21.15.104:6833/1445164438","heartbeat_front_addr":"172.21.15.104:6831/1445164438","state":["exists","up"]},{"osd":4,"uuid":"a2730a7e-cb82-4f02-bf35-56d2e7ac3544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6801","nonce":4199538417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6803","nonce":4199538417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6807","nonce":4199538417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6805","nonce":4199538417}]},"public_addr":"172.21.15.135:6801/4199538417","cluster_addr":"172.21.15.135:6803/4199538417","heartbeat_back_addr":"172.21.15.135:6807/4199538417","heartbeat_front_addr":"172.21.15.135:6805/4199538417","state":["exists","up"]},{"osd":5,"uuid":"3bdbfd71-e400-4c27-800d-a9ab591d3d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6809","nonce":2417530730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6811","nonce":2417530730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6815","nonce":2417530730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6813","nonce":2417530730}]},"public_addr":"172.21.15.135:6809/2417530730","cluster_addr":"172.21.15.135:6811/2417530730","heartbeat_back_addr":"172.21.15.135:6815/2417530730","heartbeat_front_addr":"172.21.15.135:6813/2417530730","state":["exists","up"]},{"osd":6,"uuid":"5518d906-ba93-46aa-a617-42f5fa9f7199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6817","nonce":3609133199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6819","nonce":3609133199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6823","nonce":3609133199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6821","nonce":3609133199}]},"public_addr":"172.21.15.135:6817/3609133199","cluster_addr":"172.21.15.135:6819/3609133199","heartbeat_back_addr":"172.21.15.135:6823/3609133199","heartbeat_front_addr":"172.21.15.135:6821/3609133199","state":["exists","up"]},{"osd":7,"uuid":"60426297-8db7-492e-abf3-6877a2118453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6825","nonce":3828876594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6827","nonce":3828876594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6831","nonce":3828876594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6829","nonce":3828876594}]},"public_addr":"172.21.15.135:6825/3828876594","cluster_addr":"172.21.15.135:6827/3828876594","heartbeat_back_addr":"172.21.15.135:6831/3828876594","heartbeat_front_addr":"172.21.15.135:6829/3828876594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:01:41.937779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:12.457963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:41.423475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:09.586648+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:30.762063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:54.360674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:17.494423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:41.415412+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1334657856":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1355674924":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6800/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6801/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/2776184345":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2760725563":"2024-04-28T00:59:17.823267+0000","172.21.15.104:0/1578173843":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2017927740":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2179354680":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1851986594":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1775137844":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6801/1107749034":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/1107749034":"2024-04-28T00:59:17.823267+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:04:46.971 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T01:02:45.309312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-27T01:04:46.971 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-27T01:04:47.183 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:04:47.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:46 smithi104 ceph-mon[21811]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:04:47.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:46 smithi104 ceph-mon[21811]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T01:04:47.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:46 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3226229931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:04:47.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:46 smithi135 ceph-mon[24080]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:04:47.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:46 smithi135 ceph-mon[24080]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T01:04:47.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:46 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3226229931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:04:48.302 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:47 smithi104 ceph-mon[21811]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T01:04:48.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:47 smithi135 ceph-mon[24080]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T01:04:48.957 INFO:teuthology.orchestra.run.smithi104.stdout:pg_num: 1 2024-04-27T01:04:49.316 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:48 smithi104 ceph-mon[21811]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:04:49.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:48 smithi135 ceph-mon[24080]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:04:49.678 INFO:tasks.cephadm:Setting up client nodes... 2024-04-27T01:04:49.678 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T01:04:50.220 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:49 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2549458815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T01:04:50.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:49 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2549458815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T01:04:51.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:50 smithi104 ceph-mon[21811]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-27T01:04:51.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:50 smithi135 ceph-mon[24080]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-27T01:04:51.769 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-04-27T01:04:51.769 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQAzTyxm+XYPLhAAiHLMZwpbuOIqQRBgqacXMw== 2024-04-27T01:04:52.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:51 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1881296294' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:04:52.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:51 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1881296294' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:04:52.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:51 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1881296294' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:04:52.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:51 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1881296294' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:04:52.413 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-27T01:04:52.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-27T01:04:52.413 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T01:04:52.449 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T01:04:52.449 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-27T01:04:52.449 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-27T01:04:52.695 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:04:53.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:52 smithi104 ceph-mon[21811]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-27T01:04:53.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:52 smithi135 ceph-mon[24080]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-27T01:04:54.563 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:04:55.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:54 smithi104 ceph-mon[21811]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T01:04:55.169 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:54 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2562855606' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T01:04:55.319 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi104.dpemzn","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3203584118},{"type":"v1","addr":"172.21.15.104:6801","nonce":3203584118}]},"active_addr":"172.21.15.104:6801/3203584118","active_change":"2024-04-27T01:00:40.946471+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14220,"name":"smithi135.nwoumo","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.104:8443/","prometheus":"http://172.21.15.104:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3632684906}]},{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":2503791716}]},{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":928689401}]},{"addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3220550483}]}]}} 2024-04-27T01:04:55.324 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-27T01:04:55.324 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-27T01:04:55.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:04:55.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:54 smithi135 ceph-mon[24080]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-27T01:04:55.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:54 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2562855606' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T01:04:55.536 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:04:57.329 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:04:57.329 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","created":"2024-04-27T00:58:55.078206+0000","modified":"2024-04-27T01:04:46.946478+0000","last_up_change":"2024-04-27T01:04:42.923598+0000","last_in_change":"2024-04-27T01:04:26.087299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:02:45.309312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff8ac495-617b-45e7-b25d-4d2ca1e47d00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6803","nonce":2841042446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6805","nonce":2841042446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6809","nonce":2841042446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6807","nonce":2841042446}]},"public_addr":"172.21.15.104:6803/2841042446","cluster_addr":"172.21.15.104:6805/2841042446","heartbeat_back_addr":"172.21.15.104:6809/2841042446","heartbeat_front_addr":"172.21.15.104:6807/2841042446","state":["exists","up"]},{"osd":1,"uuid":"e289d866-5a32-4b41-abef-01470666e8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6811","nonce":242749489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6813","nonce":242749489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6817","nonce":242749489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6815","nonce":242749489}]},"public_addr":"172.21.15.104:6811/242749489","cluster_addr":"172.21.15.104:6813/242749489","heartbeat_back_addr":"172.21.15.104:6817/242749489","heartbeat_front_addr":"172.21.15.104:6815/242749489","state":["exists","up"]},{"osd":2,"uuid":"91841358-2470-4ecb-a2a3-3d4d56137cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6819","nonce":1143353348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6821","nonce":1143353348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6825","nonce":1143353348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6823","nonce":1143353348}]},"public_addr":"172.21.15.104:6819/1143353348","cluster_addr":"172.21.15.104:6821/1143353348","heartbeat_back_addr":"172.21.15.104:6825/1143353348","heartbeat_front_addr":"172.21.15.104:6823/1143353348","state":["exists","up"]},{"osd":3,"uuid":"5ae4ed45-850e-46b9-8648-2c48f2d38316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6827","nonce":1445164438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6829","nonce":1445164438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6833","nonce":1445164438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6831","nonce":1445164438}]},"public_addr":"172.21.15.104:6827/1445164438","cluster_addr":"172.21.15.104:6829/1445164438","heartbeat_back_addr":"172.21.15.104:6833/1445164438","heartbeat_front_addr":"172.21.15.104:6831/1445164438","state":["exists","up"]},{"osd":4,"uuid":"a2730a7e-cb82-4f02-bf35-56d2e7ac3544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6801","nonce":4199538417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6803","nonce":4199538417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6807","nonce":4199538417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6805","nonce":4199538417}]},"public_addr":"172.21.15.135:6801/4199538417","cluster_addr":"172.21.15.135:6803/4199538417","heartbeat_back_addr":"172.21.15.135:6807/4199538417","heartbeat_front_addr":"172.21.15.135:6805/4199538417","state":["exists","up"]},{"osd":5,"uuid":"3bdbfd71-e400-4c27-800d-a9ab591d3d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6809","nonce":2417530730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6811","nonce":2417530730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6815","nonce":2417530730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6813","nonce":2417530730}]},"public_addr":"172.21.15.135:6809/2417530730","cluster_addr":"172.21.15.135:6811/2417530730","heartbeat_back_addr":"172.21.15.135:6815/2417530730","heartbeat_front_addr":"172.21.15.135:6813/2417530730","state":["exists","up"]},{"osd":6,"uuid":"5518d906-ba93-46aa-a617-42f5fa9f7199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6817","nonce":3609133199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6819","nonce":3609133199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6823","nonce":3609133199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6821","nonce":3609133199}]},"public_addr":"172.21.15.135:6817/3609133199","cluster_addr":"172.21.15.135:6819/3609133199","heartbeat_back_addr":"172.21.15.135:6823/3609133199","heartbeat_front_addr":"172.21.15.135:6821/3609133199","state":["exists","up"]},{"osd":7,"uuid":"60426297-8db7-492e-abf3-6877a2118453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6825","nonce":3828876594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6827","nonce":3828876594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6831","nonce":3828876594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6829","nonce":3828876594}]},"public_addr":"172.21.15.135:6825/3828876594","cluster_addr":"172.21.15.135:6827/3828876594","heartbeat_back_addr":"172.21.15.135:6831/3828876594","heartbeat_front_addr":"172.21.15.135:6829/3828876594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:01:41.937779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:12.457963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:41.423475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:09.586648+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:30.762063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:54.360674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:17.494423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:41.415412+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1334657856":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1355674924":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6800/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6801/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/2776184345":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2760725563":"2024-04-28T00:59:17.823267+0000","172.21.15.104:0/1578173843":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2017927740":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2179354680":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1851986594":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1775137844":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6801/1107749034":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/1107749034":"2024-04-28T00:59:17.823267+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:04:57.330 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:56 smithi104 ceph-mon[21811]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-27T01:04:57.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:56 smithi135 ceph-mon[24080]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-27T01:04:58.053 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-27T01:04:58.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:04:58.267 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:04:58.323 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2034902696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:04:58.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2034902696' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:04:59.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:04:58 smithi135 ceph-mon[24080]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-04-27T01:04:59.381 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:04:58 smithi104 ceph-mon[21811]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-04-27T01:05:00.062 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:05:00.062 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":60,"fsid":"38f5b578-0431-11ef-bc93-c7b262605968","created":"2024-04-27T00:58:55.078206+0000","modified":"2024-04-27T01:04:46.946478+0000","last_up_change":"2024-04-27T01:04:42.923598+0000","last_in_change":"2024-04-27T01:04:26.087299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:02:45.309312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff8ac495-617b-45e7-b25d-4d2ca1e47d00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6803","nonce":2841042446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6805","nonce":2841042446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6809","nonce":2841042446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":2841042446},{"type":"v1","addr":"172.21.15.104:6807","nonce":2841042446}]},"public_addr":"172.21.15.104:6803/2841042446","cluster_addr":"172.21.15.104:6805/2841042446","heartbeat_back_addr":"172.21.15.104:6809/2841042446","heartbeat_front_addr":"172.21.15.104:6807/2841042446","state":["exists","up"]},{"osd":1,"uuid":"e289d866-5a32-4b41-abef-01470666e8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6811","nonce":242749489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6813","nonce":242749489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6817","nonce":242749489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":242749489},{"type":"v1","addr":"172.21.15.104:6815","nonce":242749489}]},"public_addr":"172.21.15.104:6811/242749489","cluster_addr":"172.21.15.104:6813/242749489","heartbeat_back_addr":"172.21.15.104:6817/242749489","heartbeat_front_addr":"172.21.15.104:6815/242749489","state":["exists","up"]},{"osd":2,"uuid":"91841358-2470-4ecb-a2a3-3d4d56137cdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6819","nonce":1143353348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6821","nonce":1143353348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6825","nonce":1143353348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1143353348},{"type":"v1","addr":"172.21.15.104:6823","nonce":1143353348}]},"public_addr":"172.21.15.104:6819/1143353348","cluster_addr":"172.21.15.104:6821/1143353348","heartbeat_back_addr":"172.21.15.104:6825/1143353348","heartbeat_front_addr":"172.21.15.104:6823/1143353348","state":["exists","up"]},{"osd":3,"uuid":"5ae4ed45-850e-46b9-8648-2c48f2d38316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6827","nonce":1445164438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6829","nonce":1445164438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6833","nonce":1445164438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1445164438},{"type":"v1","addr":"172.21.15.104:6831","nonce":1445164438}]},"public_addr":"172.21.15.104:6827/1445164438","cluster_addr":"172.21.15.104:6829/1445164438","heartbeat_back_addr":"172.21.15.104:6833/1445164438","heartbeat_front_addr":"172.21.15.104:6831/1445164438","state":["exists","up"]},{"osd":4,"uuid":"a2730a7e-cb82-4f02-bf35-56d2e7ac3544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6801","nonce":4199538417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6803","nonce":4199538417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6807","nonce":4199538417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":4199538417},{"type":"v1","addr":"172.21.15.135:6805","nonce":4199538417}]},"public_addr":"172.21.15.135:6801/4199538417","cluster_addr":"172.21.15.135:6803/4199538417","heartbeat_back_addr":"172.21.15.135:6807/4199538417","heartbeat_front_addr":"172.21.15.135:6805/4199538417","state":["exists","up"]},{"osd":5,"uuid":"3bdbfd71-e400-4c27-800d-a9ab591d3d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6809","nonce":2417530730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6811","nonce":2417530730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6815","nonce":2417530730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":2417530730},{"type":"v1","addr":"172.21.15.135:6813","nonce":2417530730}]},"public_addr":"172.21.15.135:6809/2417530730","cluster_addr":"172.21.15.135:6811/2417530730","heartbeat_back_addr":"172.21.15.135:6815/2417530730","heartbeat_front_addr":"172.21.15.135:6813/2417530730","state":["exists","up"]},{"osd":6,"uuid":"5518d906-ba93-46aa-a617-42f5fa9f7199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6817","nonce":3609133199}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6819","nonce":3609133199}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6823","nonce":3609133199}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3609133199},{"type":"v1","addr":"172.21.15.135:6821","nonce":3609133199}]},"public_addr":"172.21.15.135:6817/3609133199","cluster_addr":"172.21.15.135:6819/3609133199","heartbeat_back_addr":"172.21.15.135:6823/3609133199","heartbeat_front_addr":"172.21.15.135:6821/3609133199","state":["exists","up"]},{"osd":7,"uuid":"60426297-8db7-492e-abf3-6877a2118453","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6825","nonce":3828876594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6827","nonce":3828876594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6831","nonce":3828876594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":3828876594},{"type":"v1","addr":"172.21.15.135:6829","nonce":3828876594}]},"public_addr":"172.21.15.135:6825/3828876594","cluster_addr":"172.21.15.135:6827/3828876594","heartbeat_back_addr":"172.21.15.135:6831/3828876594","heartbeat_front_addr":"172.21.15.135:6829/3828876594","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:01:41.937779+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:12.457963+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:02:41.423475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:09.586648+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:30.762063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:03:54.360674+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:17.494423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:04:41.415412+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.104:6801/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1334657856":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1355674924":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6800/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:6801/4207164935":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/2776184345":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/2125718528":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2760725563":"2024-04-28T00:59:17.823267+0000","172.21.15.104:0/1578173843":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2017927740":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/2179354680":"2024-04-28T01:00:40.946184+0000","172.21.15.104:0/1851986594":"2024-04-28T00:59:46.838179+0000","172.21.15.104:0/1775137844":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6801/1107749034":"2024-04-28T00:59:17.823267+0000","172.21.15.104:6800/1107749034":"2024-04-28T00:59:17.823267+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:05:00.806 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-27T01:05:00.807 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-27T01:05:00.807 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-27T01:05:00.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-27T01:05:00.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-27T01:05:00.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-27T01:05:00.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-27T01:05:00.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-27T01:05:01.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:00 smithi104 ceph-mon[21811]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-27T01:05:01.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:00 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1639615493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:05:01.237 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.242 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.243 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.244 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.244 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.244 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.248 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.249 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:00 smithi135 ceph-mon[24080]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-27T01:05:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:00 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1639615493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:05:03.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:02 smithi104 ceph-mon[21811]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:03.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:02 smithi135 ceph-mon[24080]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:05.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:04 smithi135 ceph-mon[24080]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:05.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:04 smithi104 ceph-mon[21811]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:07.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:07 smithi104 ceph-mon[21811]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:07.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:07 smithi135 ceph-mon[24080]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:09.262 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:09 smithi104 ceph-mon[21811]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:09.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:09 smithi135 ceph-mon[24080]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:11 smithi104 ceph-mon[21811]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:11.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:11 smithi135 ceph-mon[24080]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:11.604 INFO:teuthology.orchestra.run.smithi104.stdout:115964117018 2024-04-27T01:05:11.604 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T01:05:12.151 INFO:teuthology.orchestra.run.smithi104.stdout:81604378656 2024-04-27T01:05:12.151 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T01:05:13.251 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:13 smithi104 ceph-mon[21811]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:13.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:13 smithi135 ceph-mon[24080]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:15.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:15 smithi135 ceph-mon[24080]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:15.398 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:15 smithi104 ceph-mon[21811]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:15.422 INFO:teuthology.orchestra.run.smithi104.stdout:171798691858 2024-04-27T01:05:15.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T01:05:16.384 INFO:teuthology.orchestra.run.smithi104.stdout:38654705709 2024-04-27T01:05:16.385 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T01:05:16.612 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:17.006 INFO:teuthology.orchestra.run.smithi104.stdout:240518168586 2024-04-27T01:05:17.006 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-27T01:05:17.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:17 smithi135 ceph-mon[24080]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:17.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:17 smithi104 ceph-mon[21811]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:17.691 INFO:teuthology.orchestra.run.smithi104.stdout:137438953496 2024-04-27T01:05:17.691 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T01:05:17.936 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:17.937 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:18.364 INFO:teuthology.orchestra.run.smithi104.stdout:206158430222 2024-04-27T01:05:18.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-27T01:05:18.547 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:18.548 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:19.348 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:19 smithi104 ceph-mon[21811]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:19.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:19 smithi135 ceph-mon[24080]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:21.276 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:21 smithi104 ceph-mon[21811]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:21.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:21 smithi135 ceph-mon[24080]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:22.949 INFO:teuthology.orchestra.run.smithi104.stdout:60129542184 2024-04-27T01:05:22.949 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T01:05:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:23 smithi104 ceph-mon[21811]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:23 smithi135 ceph-mon[24080]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:23.798 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:23.967 INFO:teuthology.orchestra.run.smithi104.stdout:38654705711 2024-04-27T01:05:24.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:24 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4114660763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T01:05:24.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:24 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4114660763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T01:05:25.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:25 smithi135 ceph-mon[24080]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:25.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:25 smithi104 ceph-mon[21811]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:25.543 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:25.724 INFO:teuthology.orchestra.run.smithi104.stdout:171798691860 2024-04-27T01:05:26.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:26 smithi104 ceph-mon[21811]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:26.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:26 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/618066644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T01:05:26.389 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:26 smithi135 ceph-mon[24080]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:26.390 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:26 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/618066644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T01:05:26.513 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:26.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705711 for osd.0 2024-04-27T01:05:26.989 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:27.474 INFO:teuthology.orchestra.run.smithi104.stdout:81604378660 2024-04-27T01:05:28.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:28 smithi135 ceph-mon[24080]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:28.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:28 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2826521607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T01:05:28.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:28 smithi104 ceph-mon[21811]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:28.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:28 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2826521607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T01:05:29.012 INFO:teuthology.orchestra.run.smithi104.stdout:206158430224 2024-04-27T01:05:29.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:29 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4027305254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T01:05:29.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:29 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3014908539' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T01:05:29.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:29 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4027305254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T01:05:29.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:29 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3014908539' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T01:05:29.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430224 for osd.6 2024-04-27T01:05:29.808 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:30.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:30 smithi135 ceph-mon[24080]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:30.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:30 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/792316504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T01:05:30.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:30 smithi104 ceph-mon[21811]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:30.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:30 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/792316504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T01:05:30.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691858 got 171798691860 for osd.5 2024-04-27T01:05:30.397 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:30.473 INFO:teuthology.orchestra.run.smithi104.stdout:240518168589 2024-04-27T01:05:31.080 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168586 got 240518168589 for osd.7 2024-04-27T01:05:31.080 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:31.152 INFO:teuthology.orchestra.run.smithi104.stdout:115964117021 2024-04-27T01:05:31.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117018 got 115964117021 for osd.3 2024-04-27T01:05:31.717 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:32 smithi104 ceph-mon[21811]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:32.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:32 smithi135 ceph-mon[24080]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:33.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378660 for osd.2 2024-04-27T01:05:33.047 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:34.365 INFO:teuthology.orchestra.run.smithi104.stdout:60129542187 2024-04-27T01:05:34.365 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:34 smithi104 ceph-mon[21811]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:34.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:34 smithi135 ceph-mon[24080]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:35.150 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:35 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1023466749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T01:05:35.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:35 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1023466749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T01:05:35.427 INFO:teuthology.orchestra.run.smithi104.stdout:137438953500 2024-04-27T01:05:35.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542184 got 60129542187 for osd.1 2024-04-27T01:05:35.667 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:36.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:36 smithi104 ceph-mon[21811]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:36.298 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:36 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/394629553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T01:05:36.324 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953496 got 137438953500 for osd.4 2024-04-27T01:05:36.325 DEBUG:teuthology.parallel:result is None 2024-04-27T01:05:36.325 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T01:05:36.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T01:05:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:36 smithi135 ceph-mon[24080]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:36 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/394629553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T01:05:36.542 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:38.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:05:38.222 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-04-27T01:05:38.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:38 smithi135 ceph-mon[24080]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:38.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:38 smithi104 ceph-mon[21811]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:38.877 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-27T01:05:37.049045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48952,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912264,"statfs":{"total":767960285184,"available":767910158336,"internally_reserved":0,"allocated":6545408,"data_stored":3572496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001983"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-27T01:04:46.956650+0000","last_change":"2024-04-27T01:04:46.956650+0000","last_active":"2024-04-27T01:04:46.956650+0000","last_peered":"2024-04-27T01:04:46.956650+0000","last_clean":"2024-04-27T01:04:46.956650+0000","last_became_active":"2024-04-27T01:04:46.956164+0000","last_became_peered":"2024-04-27T01:04:46.956164+0000","last_unstale":"2024-04-27T01:04:46.956650+0000","last_undegraded":"2024-04-27T01:04:46.956650+0000","last_fullsized":"2024-04-27T01:04:46.956650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:02:45.889312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:02:45.889312+0000","last_clean_scrub_stamp":"2024-04-27T01:02:45.889312+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:40.901745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]},{"osd":6,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60299999999999998}]},{"osd":1,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.63300000000000001}]},{"osd":3,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.442}]},{"osd":5,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.502}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33600000000000002,"15min":0.315},"min":{"1min":0.21199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Apr 27 01:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.378,"15min":0.371},"min":{"1min":0.25900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.39100000000000001,"15min":0.38400000000000001},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.36499999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.432,"15min":0.435},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.436,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":1.018,"15min":1.0880000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.80900000000000005,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.434,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Apr 27 01:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.503},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.46500000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.379,"5min":0.36399999999999999,"15min":0.34399999999999997},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.55200000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Apr 27 01:04:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.41499999999999998},"min":{"1min":0.34799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42899999999999999,"15min":0.434},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.47699999999999998,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Sat Apr 27 01:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.79300000000000004,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Apr 27 01:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70699999999999996}]},{"osd":6,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Sat Apr 27 01:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.309,"5min":0.254,"15min":0.254},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.39700000000000002},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Apr 27 01:05:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.34799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.621},{"interface":"front","average":{"1min":0.495,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Apr 27 01:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":14.427,"5min":14.427,"15min":14.427},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Apr 27 01:04:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":15.098000000000001,"5min":15.098000000000001,"15min":15.098000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.245,"15min":0.245},"max":{"1min":0.50600000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.41399999999999998,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51700000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.25800000000000001}]},{"osd":1,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.44800000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46899999999999997,"5min":0.752,"15min":0.752},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.50700000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.40600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.502,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.312}]},{"osd":4,"last update":"Sat Apr 27 01:04:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.38400000000000001}]},{"osd":5,"last update":"Sat Apr 27 01:04:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.501}]},{"osd":6,"last update":"Sat Apr 27 01:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":19.164999999999999,"5min":19.164999999999999,"15min":19.164999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Apr 27 01:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.68700000000000006}]},{"osd":2,"last update":"Sat Apr 27 01:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":19.193999999999999,"5min":19.193999999999999,"15min":19.193999999999999},"last":0.73799999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:04:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":19.216000000000001,"5min":19.216000000000001,"15min":19.216000000000001},"last":0.71399999999999997}]},{"osd":5,"last update":"Sat Apr 27 01:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.624}]},{"osd":6,"last update":"Sat Apr 27 01:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.5},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36399999999999999}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.374},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T01:05:38.878 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T01:05:39.089 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:39.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:39 smithi135 ceph-mon[24080]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:39.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:39 smithi104 ceph-mon[21811]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:40.706 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:05:40.708 INFO:teuthology.orchestra.run.smithi104.stderr:dumped all 2024-04-27T01:05:41.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:40 smithi104 ceph-mon[21811]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:41.354 INFO:teuthology.orchestra.run.smithi104.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-27T01:05:39.049515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48952,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912264,"statfs":{"total":767960285184,"available":767910158336,"internally_reserved":0,"allocated":6545408,"data_stored":3572496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002022"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-27T01:04:46.956650+0000","last_change":"2024-04-27T01:04:46.956650+0000","last_active":"2024-04-27T01:04:46.956650+0000","last_peered":"2024-04-27T01:04:46.956650+0000","last_clean":"2024-04-27T01:04:46.956650+0000","last_became_active":"2024-04-27T01:04:46.956164+0000","last_became_peered":"2024-04-27T01:04:46.956164+0000","last_unstale":"2024-04-27T01:04:46.956650+0000","last_undegraded":"2024-04-27T01:04:46.956650+0000","last_fullsized":"2024-04-27T01:04:46.956650+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:02:45.889312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:02:45.889312+0000","last_clean_scrub_stamp":"2024-04-27T01:02:45.889312+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:50:40.901745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.62},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46999999999999997}]},{"osd":1,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.66600000000000004}]},{"osd":4,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.63300000000000001}]},{"osd":5,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.68000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.33600000000000002,"15min":0.315},"min":{"1min":0.21199999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.34200000000000003,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Apr 27 01:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.378,"15min":0.371},"min":{"1min":0.25900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.39100000000000001,"15min":0.38400000000000001},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.36499999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.432,"15min":0.435},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58399999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.436,"15min":0.437},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58499999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":1.018,"15min":1.0880000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.80900000000000005,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.434,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Apr 27 01:05:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.503},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35199999999999998,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.46500000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.379,"5min":0.36399999999999999,"15min":0.34399999999999997},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.55200000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Apr 27 01:04:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.42299999999999999,"15min":0.41499999999999998},"min":{"1min":0.34799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42899999999999999,"15min":0.434},"min":{"1min":0.30199999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.47699999999999998,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Sat Apr 27 01:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.79300000000000004,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Apr 27 01:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70699999999999996}]},{"osd":6,"last update":"Sat Apr 27 01:05:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Sat Apr 27 01:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36899999999999999,"15min":0.36299999999999999},"min":{"1min":0.309,"5min":0.254,"15min":0.254},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.40500000000000003,"15min":0.39700000000000002},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Apr 27 01:05:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.34799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.621},{"interface":"front","average":{"1min":0.495,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33400000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Apr 27 01:04:37 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":14.427,"5min":14.427,"15min":14.427},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Apr 27 01:04:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":15.098000000000001,"5min":15.098000000000001,"15min":15.098000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.67100000000000004}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.69099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36599999999999999,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.245,"15min":0.245},"max":{"1min":0.50600000000000001,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.41399999999999998,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.51700000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.44800000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.216},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46899999999999997,"5min":0.752,"15min":0.752},"last":0.28199999999999997}]},{"osd":2,"last update":"Sat Apr 27 01:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.50700000000000001,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.315},{"interface":"front","average":{"1min":0.371,"5min":0.40100000000000002,"15min":0.40600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.502,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32700000000000001}]},{"osd":4,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.38400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55000000000000004}]},{"osd":5,"last update":"Sat Apr 27 01:04:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Apr 27 01:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953501,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5796,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739356,"statfs":{"total":95995035648,"available":95989100544,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.40899999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.85199999999999998,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.2010000000000001,"15min":1.2949999999999999},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.83999999999999997,"5min":19.164999999999999,"15min":19.164999999999999},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.35899999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.82099999999999995,"5min":1.706,"15min":1.706},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.39100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.439,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.751,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":1.1499999999999999,"15min":1.2390000000000001},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81299999999999994,"5min":19.193999999999999,"15min":19.193999999999999},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Apr 27 01:05:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.435,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.1910000000000001,"15min":1.2869999999999999},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":19.216000000000001,"15min":19.216000000000001},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Apr 27 01:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Apr 27 01:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.5},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Apr 27 01:04:58 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.36399999999999999}]},{"osd":6,"last update":"Sat Apr 27 01:05:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.374},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T01:05:41.355 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T01:05:41.355 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T01:05:41.356 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T01:05:41.356 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T01:05:41.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:40 smithi135 ceph-mon[24080]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:41.567 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/mon.smithi104/config 2024-04-27T01:05:41.939 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:41 smithi104 ceph-mon[21811]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:41.939 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:05:41.939 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:05:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:41 smithi135 ceph-mon[24080]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:42.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:05:42.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:05:43.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:42 smithi104 ceph-mon[21811]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:43.324 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:05:43.324 INFO:teuthology.orchestra.run.smithi104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T01:05:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:42 smithi135 ceph-mon[24080]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:43.953 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:43 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2784275150' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T01:05:43.981 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T01:05:43.981 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T01:05:43.981 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T01:05:43.991 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:05:43.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-27T01:05:44.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:43 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2784275150' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T01:05:45.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:44 smithi104 ceph-mon[21811]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:45.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:45.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:45.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:44 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:45.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:44 smithi135 ceph-mon[24080]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:45.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:45.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:45.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:44 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:45.924 INFO:teuthology.orchestra.run.smithi104.stdout:Backend: cephadm 2024-04-27T01:05:45.925 INFO:teuthology.orchestra.run.smithi104.stdout:Available: Yes 2024-04-27T01:05:45.925 INFO:teuthology.orchestra.run.smithi104.stdout:Paused: No 2024-04-27T01:05:46.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:45 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:05:46.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:45 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:05:46.656 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-27T01:05:47.217 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:46 smithi104 ceph-mon[21811]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:47.217 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:46 smithi104 ceph-mon[21811]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:47.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:46 smithi135 ceph-mon[24080]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:47.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:46 smithi135 ceph-mon[24080]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:48.659 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (4m) 2m ago 5m 17.6M - ba2b418f427c a5811d92dc4f 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (4m) 67s ago 4m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (3m) 2m ago 5m 45.3M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (4m) 67s ago 4m 414M - 17.2.0 e1d6a67b021e a07c1bcb86e1 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (6m) 2m ago 6m 41.7M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (4m) 67s ago 4m 42.0M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 2m ago 5m 17.2M - 1dbe0e931976 10b041438edc 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (4m) 67s ago 4m 18.7M - 1dbe0e931976 10a43cc6931b 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (4m) 2m ago 4m 37.0M 3216M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:05:48.660 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (3m) 2m ago 3m 34.0M 3216M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (3m) 2m ago 3m 32.0M 3216M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (2m) 2m ago 2m 24.8M 3216M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (2m) 67s ago 2m 33.2M 3984M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (118s) 67s ago 118s 32.9M 3984M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (95s) 67s ago 95s 33.1M 3984M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (71s) 67s ago 71s 9.92M 3984M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:05:48.661 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (4m) 2m ago 4m 52.8M - 514e6a882f6e e619d633de42 2024-04-27T01:05:49.260 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:48 smithi104 ceph-mon[21811]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:49.285 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-27T01:05:49.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:48 smithi135 ceph-mon[24080]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:50.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:49 smithi104 ceph-mon[21811]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:50.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:49 smithi135 ceph-mon[24080]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:crash 2/2 2m ago 6m * 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:mgr 2/2 2m ago 6m count:2 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:mon 2/2 2m ago 5m smithi104:172.21.15.104=smithi104;smithi135:172.21.15.135=smithi135;count:2 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:osd 8 2m ago - 2024-04-27T01:05:51.340 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-27T01:05:51.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:50 smithi104 ceph-mon[21811]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:51.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:50 smithi135 ceph-mon[24080]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:52.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-27T01:05:53.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:52 smithi104 ceph-mon[21811]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:53.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:52 smithi104 ceph-mon[21811]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:53.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:52 smithi135 ceph-mon[24080]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:53.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:52 smithi135 ceph-mon[24080]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:53.939 INFO:teuthology.orchestra.run.smithi104.stdout:HOST ADDR LABELS STATUS 2024-04-27T01:05:53.939 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104 172.21.15.104 2024-04-27T01:05:53.940 INFO:teuthology.orchestra.run.smithi104.stdout:smithi135 172.21.15.135 2024-04-27T01:05:53.940 INFO:teuthology.orchestra.run.smithi104.stdout:2 hosts in cluster 2024-04-27T01:05:54.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-27T01:05:55.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:54 smithi104 ceph-mon[21811]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:55.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:54 smithi104 ceph-mon[21811]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:55.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:54 smithi135 ceph-mon[24080]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:55.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:54 smithi135 ceph-mon[24080]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:56.493 INFO:teuthology.orchestra.run.smithi104.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-27T01:05:56.493 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 400G LVM detected, locked 2024-04-27T01:05:56.493 INFO:teuthology.orchestra.run.smithi104.stdout:smithi135 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620401BA400BGN 400G LVM detected, locked 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:56 smithi104 ceph-mon[21811]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:57.155 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T01:05:57.167 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:05:57.167 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-27T01:05:57.191 INFO:teuthology.orchestra.run.smithi104.stderr:+ systemctl stop nfs-server 2024-04-27T01:05:57.199 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi135.front.sepia.ceph.com 2024-04-27T01:05:57.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-27T01:05:57.227 INFO:teuthology.orchestra.run.smithi135.stderr:+ systemctl stop nfs-server 2024-04-27T01:05:57.236 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T01:05:57.243 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:05:57.243 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create foofs' 2024-04-27T01:05:57.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:57 smithi135 ceph-mon[24080]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:58.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:58 smithi104 ceph-mon[21811]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:58.269 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:58 smithi135 ceph-mon[24080]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:59.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:05:59 smithi104 ceph-mon[21811]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:05:59.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:05:59 smithi135 ceph-mon[24080]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:00.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:00 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-27T01:06:00.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:00 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-27T01:06:01.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:01 smithi135 ceph-mon[24080]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:01 smithi135 ceph-mon[24080]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:01 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-27T01:06:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:01 smithi135 ceph-mon[24080]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T01:06:01.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:01 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-mon[21811]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-mon[21811]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-mon[21811]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-27T01:06:01.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:01 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[21807]: 2024-04-27T01:06:01.039+0000 7fdea60f0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:06:01.727 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-27T01:06:01.736 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-27T01:06:01.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T01:06:02.248 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: fsmap foofs:0 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:06:02.250 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T01:06:02.250 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: fsmap foofs:0 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:02.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:06:02.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T01:06:02.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:03.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:03 smithi104 ceph-mon[21811]: Saving service mds.foofs spec with placement count:2 2024-04-27T01:06:03.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:03 smithi104 ceph-mon[21811]: pgmap v211: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:03.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:03 smithi104 ceph-mon[21811]: Deploying daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:06:03.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:03 smithi104 ceph-mon[21811]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T01:06:03.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:03 smithi135 ceph-mon[24080]: Saving service mds.foofs spec with placement count:2 2024-04-27T01:06:03.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:03 smithi135 ceph-mon[24080]: pgmap v211: 65 pgs: 11 creating+peering, 1 active+clean, 53 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:03.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:03 smithi135 ceph-mon[24080]: Deploying daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:06:03.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:03 smithi135 ceph-mon[24080]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T01:06:04.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:04 smithi104 ceph-mon[21811]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T01:06:04.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:04 smithi104 ceph-mon[21811]: pgmap v214: 65 pgs: 11 creating+peering, 15 active+clean, 39 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:04.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:04 smithi135 ceph-mon[24080]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T01:06:04.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:04 smithi135 ceph-mon[24080]: pgmap v214: 65 pgs: 11 creating+peering, 15 active+clean, 39 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:04.905 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:06:04.905 INFO:teuthology.orchestra.run.smithi104.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T00:59:35.829242Z", "last_refresh": "2024-04-27T01:03:19.463545Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T00:59:30.302885Z", "last_refresh": "2024-04-27T01:03:19.463623Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T00:59:33.305126Z", "last_refresh": "2024-04-27T01:03:19.463687Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T01:06:01.061638Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T01:06:01.052571Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T00:59:28.673051Z", "last_refresh": "2024-04-27T01:03:19.463473Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi104:172.21.15.104=smithi104", "smithi135:172.21.15.135=smithi135"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T01:00:19.253264Z", "last_refresh": "2024-04-27T01:03:19.463326Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T00:59:34.600681Z", "last_refresh": "2024-04-27T01:03:19.463748Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T01:03:19.463887Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T00:59:32.016734Z", "last_refresh": "2024-04-27T01:03:19.463809Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T01:06:05.661 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-27T01:06:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:05 smithi104 ceph-mon[21811]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:05 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:05 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:06:06.326 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:05 smithi135 ceph-mon[24080]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:06.326 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:05 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:06.326 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:05 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:06:06.662 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T01:06:06.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: pgmap v215: 65 pgs: 11 creating+peering, 46 active+clean, 8 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: Deploying daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: mds.? [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] up:boot 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: daemon mds.foofs.smithi104.wfnrbv assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: Cluster is now healthy 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: fsmap foofs:0 1 up:standby 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:creating} 2024-04-27T01:06:06.977 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:06 smithi104 ceph-mon[21811]: daemon mds.foofs.smithi104.wfnrbv is now active in filesystem foofs as rank 0 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: pgmap v215: 65 pgs: 11 creating+peering, 46 active+clean, 8 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: Deploying daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: mds.? [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] up:boot 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: daemon mds.foofs.smithi104.wfnrbv assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: Cluster is now healthy 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: fsmap foofs:0 1 up:standby 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:creating} 2024-04-27T01:06:07.344 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:06 smithi135 ceph-mon[24080]: daemon mds.foofs.smithi104.wfnrbv is now active in filesystem foofs as rank 0 2024-04-27T01:06:08.228 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:07 smithi135 ceph-mon[24080]: mds.? [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] up:active 2024-04-27T01:06:08.229 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:07 smithi135 ceph-mon[24080]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 2024-04-27T01:06:08.259 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:07 smithi104 ceph-mon[21811]: mds.? [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] up:active 2024-04-27T01:06:08.260 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:07 smithi104 ceph-mon[21811]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 2024-04-27T01:06:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:06:08.830 INFO:teuthology.orchestra.run.smithi104.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T00:59:35.829242Z", "last_refresh": "2024-04-27T01:03:19.463545Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T00:59:30.302885Z", "last_refresh": "2024-04-27T01:03:19.463623Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T00:59:33.305126Z", "last_refresh": "2024-04-27T01:03:19.463687Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T01:06:01.061638Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T01:06:01.052571Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T00:59:28.673051Z", "last_refresh": "2024-04-27T01:03:19.463473Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi104:172.21.15.104=smithi104", "smithi135:172.21.15.135=smithi135"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T01:00:19.253264Z", "last_refresh": "2024-04-27T01:03:19.463326Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T00:59:34.600681Z", "last_refresh": "2024-04-27T01:03:19.463748Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T01:03:19.463887Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T00:59:32.016734Z", "last_refresh": "2024-04-27T01:03:19.463809Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T01:06:09.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:08 smithi104 ceph-mon[21811]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:09.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:09.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:09.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:09.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:08 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:09.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:08 smithi135 ceph-mon[24080]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:06:09.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:09.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:09.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:09.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:08 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:09.652 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-27T01:06:10.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:09 smithi135 ceph-mon[24080]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:10.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:09 smithi135 ceph-mon[24080]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:boot 2024-04-27T01:06:10.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:09 smithi135 ceph-mon[24080]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:06:10.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:09 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:06:10.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:09 smithi135 ceph-mon[24080]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:06:10.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:09 smithi104 ceph-mon[21811]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:10.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:09 smithi104 ceph-mon[21811]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:boot 2024-04-27T01:06:10.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:09 smithi104 ceph-mon[21811]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:06:10.398 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:09 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:06:10.398 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:09 smithi104 ceph-mon[21811]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:06:10.652 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T01:06:11.235 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:10 smithi104 ceph-mon[21811]: pgmap v217: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 2 op/s 2024-04-27T01:06:11.235 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:10 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:11.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:10 smithi135 ceph-mon[24080]: pgmap v217: 65 pgs: 65 active+clean; 578 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 2 op/s 2024-04-27T01:06:11.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:10 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:12.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:12 smithi104 ceph-mon[21811]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T01:06:12.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:12 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:12.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:12 smithi135 ceph-mon[24080]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-27T01:06:12.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:12 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:13.410 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:06:13.411 INFO:teuthology.orchestra.run.smithi104.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T00:59:35.829242Z", "last_refresh": "2024-04-27T01:03:19.463545Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T00:59:30.302885Z", "last_refresh": "2024-04-27T01:03:19.463623Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T00:59:33.305126Z", "last_refresh": "2024-04-27T01:03:19.463687Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T01:06:01.061638Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T01:06:01.052571Z", "last_refresh": "2024-04-27T01:06:10.397114Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T00:59:28.673051Z", "last_refresh": "2024-04-27T01:03:19.463473Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi104:172.21.15.104=smithi104", "smithi135:172.21.15.135=smithi135"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T01:00:19.253264Z", "last_refresh": "2024-04-27T01:03:19.463326Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T00:59:34.600681Z", "last_refresh": "2024-04-27T01:03:19.463748Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T01:03:19.463887Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T00:59:32.016734Z", "last_refresh": "2024-04-27T01:03:19.463809Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T01:06:14.101 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-27T01:06:14.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:14 smithi104 ceph-mon[21811]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-27T01:06:14.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:14 smithi104 ceph-mon[21811]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:14.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:14 smithi135 ceph-mon[24080]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-27T01:06:14.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:14 smithi135 ceph-mon[24080]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:15.102 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:16 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:16.868 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:16 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:17.223 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:06:17.223 INFO:teuthology.orchestra.run.smithi104.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T00:59:35.829242Z", "last_refresh": "2024-04-27T01:06:15.605575Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T00:59:30.302885Z", "last_refresh": "2024-04-27T01:06:10.396295Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-27T00:59:33.305126Z", "last_refresh": "2024-04-27T01:06:15.605705Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-27T01:06:01.061638Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T01:06:01.052571Z", "last_refresh": "2024-04-27T01:06:10.397114Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T00:59:28.673051Z", "last_refresh": "2024-04-27T01:06:10.396488Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi104:172.21.15.104=smithi104", "smithi135:172.21.15.135=smithi135"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T01:00:19.253264Z", "last_refresh": "2024-04-27T01:06:10.396612Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T00:59:34.600681Z", "last_refresh": "2024-04-27T01:06:10.396719Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-27T01:06:10.396803Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T00:59:32.016734Z", "last_refresh": "2024-04-27T01:06:15.605827Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T01:06:18.001 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-27T01:06:18.001 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T01:06:18.012 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:06:18.012 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-27T01:06:18.867 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:18 smithi135 ceph-mon[24080]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:18.867 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:18 smithi135 ceph-mon[24080]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:18.917 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:18 smithi104 ceph-mon[21811]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:18.918 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:18 smithi104 ceph-mon[21811]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:21.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:20 smithi135 ceph-mon[24080]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:21.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:20 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-27T01:06:21.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:20 smithi104 ceph-mon[21811]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-27T01:06:21.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:20 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-27T01:06:21.972 INFO:teuthology.orchestra.run.smithi104.stdout:NFS Cluster Created Successfully 2024-04-27T01:06:22.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:21 smithi104 ceph-mon[21811]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:22.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:21 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-27T01:06:22.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:21 smithi104 ceph-mon[21811]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T01:06:22.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:21 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-27T01:06:22.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:21 smithi135 ceph-mon[24080]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:22.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:21 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-27T01:06:22.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:21 smithi135 ceph-mon[24080]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T01:06:22.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:21 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-27T01:06:22.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: pgmap v224: 97 pgs: 2 creating+peering, 30 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: Saving service nfs.foo spec with placement count:2 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.021 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:06:23.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:22 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: pgmap v224: 97 pgs: 2 creating+peering, 30 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: Saving service nfs.foo spec with placement count:2 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:06:23.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:06:23.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:06:23.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:06:23.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:06:23.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:22 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:24.022 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih 2024-04-27T01:06:24.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T01:06:24.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: Rados config object exists: conf-nfs.foo 2024-04-27T01:06:24.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih-rgw 2024-04-27T01:06:24.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: Deploying daemon nfs.foo.0.0.smithi104.yxqnih on smithi104 2024-04-27T01:06:24.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:23 smithi104 ceph-mon[21811]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T01:06:24.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih 2024-04-27T01:06:24.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T01:06:24.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: Rados config object exists: conf-nfs.foo 2024-04-27T01:06:24.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih-rgw 2024-04-27T01:06:24.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: Deploying daemon nfs.foo.0.0.smithi104.yxqnih on smithi104 2024-04-27T01:06:24.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:23 smithi135 ceph-mon[24080]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T01:06:25.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:24 smithi104 ceph-mon[21811]: pgmap v227: 97 pgs: 2 creating+peering, 14 unknown, 81 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T01:06:25.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:24 smithi104 ceph-mon[21811]: mgrmap e20: smithi104.dpemzn(active, since 5m), standbys: smithi135.nwoumo 2024-04-27T01:06:25.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:24 smithi135 ceph-mon[24080]: pgmap v227: 97 pgs: 2 creating+peering, 14 unknown, 81 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T01:06:25.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:24 smithi135 ceph-mon[24080]: mgrmap e20: smithi104.dpemzn(active, since 5m), standbys: smithi135.nwoumo 2024-04-27T01:06:25.927 INFO:teuthology.orchestra.run.smithi104.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-27T01:06:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-27T01:06:25.928 INFO:teuthology.orchestra.run.smithi104.stderr:Error EINVAL: invalid command 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout: "bind": "/fake", 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout: "fs": "foofs", 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster": "foo", 2024-04-27T01:06:26.291 INFO:teuthology.orchestra.run.smithi104.stdout: "mode": "RW" 2024-04-27T01:06:26.292 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:06:26.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-27T01:06:27.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: pgmap v228: 97 pgs: 2 creating+peering, 5 unknown, 90 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-27T01:06:27.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-27T01:06:27.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: pgmap v228: 97 pgs: 2 creating+peering, 5 unknown, 90 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-27T01:06:27.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:27.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:06:27.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-27T01:06:27.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:06:27.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:06:27.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:27.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-27T01:06:27.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-27T01:06:28.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:27 smithi104 ceph-mon[21811]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:28.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:27 smithi104 ceph-mon[21811]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs 2024-04-27T01:06:28.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:27 smithi104 ceph-mon[21811]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T01:06:28.269 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:27 smithi135 ceph-mon[24080]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:28.269 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:27 smithi135 ceph-mon[24080]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs 2024-04-27T01:06:28.269 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:27 smithi135 ceph-mon[24080]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T01:06:29.232 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:28 smithi135 ceph-mon[24080]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 334 B/s wr, 0 op/s 2024-04-27T01:06:29.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:28 smithi104 ceph-mon[21811]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 334 B/s wr, 0 op/s 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:06:30.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:29 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:30.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:30.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:06:30.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:06:30.417 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:06:30.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:06:30.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:29 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:31.297 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:30 smithi135 ceph-mon[24080]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:06:31.298 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:30 smithi135 ceph-mon[24080]: Rados config object exists: conf-nfs.foo 2024-04-27T01:06:31.298 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:30 smithi135 ceph-mon[24080]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs-rgw 2024-04-27T01:06:31.298 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:30 smithi135 ceph-mon[24080]: Deploying daemon nfs.foo.1.0.smithi135.dbfshs on smithi135 2024-04-27T01:06:31.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:30 smithi104 ceph-mon[21811]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:06:31.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:30 smithi104 ceph-mon[21811]: Rados config object exists: conf-nfs.foo 2024-04-27T01:06:31.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:30 smithi104 ceph-mon[21811]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs-rgw 2024-04-27T01:06:31.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:30 smithi104 ceph-mon[21811]: Deploying daemon nfs.foo.1.0.smithi135.dbfshs on smithi135 2024-04-27T01:06:32.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:31 smithi135 ceph-mon[24080]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:32.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:32.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:32.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:32.273 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:32.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:31 smithi104 ceph-mon[21811]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:32.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:32.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:32.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:32.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:33.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:32 smithi104 ceph-mon[21811]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:06:33.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:32 smithi104 ceph-mon[21811]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:33.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:32 smithi135 ceph-mon[24080]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:06:33.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:32 smithi135 ceph-mon[24080]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:34.034 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:33 smithi135 ceph-mon[24080]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:34.369 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:33 smithi104 ceph-mon[21811]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:35.237 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:34 smithi104 ceph-mon[21811]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-27T01:06:35.237 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:35.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:34 smithi135 ceph-mon[24080]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-27T01:06:35.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:36.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:35 smithi104 ceph-mon[21811]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:35 smithi135 ceph-mon[24080]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:36.819 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-27T01:06:37.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:37.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:37.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:37.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:06:37.579 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T01:06:37.588 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:06:37.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-27T01:06:37.613 INFO:teuthology.orchestra.run.smithi104.stderr:+ mkdir /mnt/foo 2024-04-27T01:06:37.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-27T01:06:37.686 INFO:teuthology.orchestra.run.smithi104.stderr:++ hostname 2024-04-27T01:06:37.687 INFO:teuthology.orchestra.run.smithi104.stderr:+ mount -t nfs smithi104:/fake /mnt/foo -o sync 2024-04-27T01:06:38.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-27T01:06:38.147 INFO:teuthology.orchestra.run.smithi104.stderr:+ echo test 2024-04-27T01:06:38.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-27T01:06:38.219 INFO:teuthology.orchestra.run.smithi104.stderr:+ sync 2024-04-27T01:06:38.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:37 smithi135 ceph-mon[24080]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:38.405 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:37 smithi104 ceph-mon[21811]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:38.406 INFO:teuthology.run_tasks:Running task parallel... 2024-04-27T01:06:38.418 INFO:teuthology.task.parallel:starting parallel... 2024-04-27T01:06:38.418 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T01:06:38.418 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T01:06:38.419 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:06:38.419 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-27T01:06:38.419 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-27T01:06:38.419 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-27T01:06:38.420 INFO:teuthology.task.exec:Executing custom commands... 2024-04-27T01:06:38.420 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:06:38.420 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-27T01:06:38.494 INFO:teuthology.orchestra.run.smithi104.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-27T01:06:38.494 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:06:38.494 INFO:teuthology.orchestra.run.smithi104.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-27T01:06:38.495 INFO:teuthology.orchestra.run.smithi104.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-27T01:06:38.595 INFO:teuthology.orchestra.run.smithi104.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-27T01:06:38.595 INFO:teuthology.orchestra.run.smithi104.stdout:releasing clients 2024-04-27T01:06:39.189 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:39 smithi104 ceph-mon[21811]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T01:06:39.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:39 smithi135 ceph-mon[24080]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-27T01:06:39.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 301 55.15 MB/sec warmup 1 sec latency 39.208 ms 2024-04-27T01:06:40.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 629 52.33 MB/sec warmup 2 sec latency 11.595 ms 2024-04-27T01:06:41.009 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:41 smithi104 ceph-mon[21811]: pgmap v235: 97 pgs: 97 active+clean; 5.0 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 414 KiB/s wr, 44 op/s 2024-04-27T01:06:41.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:41 smithi135 ceph-mon[24080]: pgmap v235: 97 pgs: 97 active+clean; 5.0 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 414 KiB/s wr, 44 op/s 2024-04-27T01:06:41.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 1354 41.42 MB/sec warmup 3 sec latency 13.626 ms 2024-04-27T01:06:42.015 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-04-27T01:06:42.015 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-27T01:06:42.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-27T01:06:42.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-27T01:06:42.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:06:42.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:42 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:06:42.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:42 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:06:42.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 1803 32.89 MB/sec warmup 4 sec latency 594.575 ms 2024-04-27T01:06:42.759 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-27T01:06:43.256 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: pgmap v236: 97 pgs: 97 active+clean; 36 MiB data, 77 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.0 MiB/s wr, 93 op/s 2024-04-27T01:06:43.257 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-04-27T01:06:43.257 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-27T01:06:43.257 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-27T01:06:43.257 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-27T01:06:43.257 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:43 smithi104 ceph-mon[21811]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T01:06:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: pgmap v236: 97 pgs: 97 active+clean; 36 MiB data, 77 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.0 MiB/s wr, 93 op/s 2024-04-27T01:06:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-04-27T01:06:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-27T01:06:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-27T01:06:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-27T01:06:43.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:43 smithi135 ceph-mon[24080]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T01:06:43.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 1803 26.31 MB/sec warmup 5 sec latency 1594.605 ms 2024-04-27T01:06:44.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:44 smithi104 ceph-mon[21811]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T01:06:44.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:44 smithi135 ceph-mon[24080]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T01:06:44.595 INFO:teuthology.orchestra.run.smithi104.stdout: 5 2368 23.27 MB/sec warmup 6 sec latency 2108.277 ms 2024-04-27T01:06:45.242 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:45 smithi104 ceph-mon[21811]: pgmap v239: 97 pgs: 97 active+clean; 73 MiB data, 115 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 9.5 MiB/s wr, 413 op/s 2024-04-27T01:06:45.242 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:45 smithi104 ceph-mon[21811]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T01:06:45.242 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:45 smithi104 ceph-mon[21811]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T01:06:45.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:45 smithi135 ceph-mon[24080]: pgmap v239: 97 pgs: 97 active+clean; 73 MiB data, 115 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 9.5 MiB/s wr, 413 op/s 2024-04-27T01:06:45.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:45 smithi135 ceph-mon[24080]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T01:06:45.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:45 smithi135 ceph-mon[24080]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T01:06:45.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 2705 21.62 MB/sec warmup 7 sec latency 665.421 ms 2024-04-27T01:06:46.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-27T01:06:46.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 3417 22.86 MB/sec warmup 8 sec latency 1020.656 ms 2024-04-27T01:06:47.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:46 smithi104 ceph-mon[21811]: pgmap v242: 97 pgs: 97 active+clean; 108 MiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 20 MiB/s wr, 1.10k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:47.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:46 smithi104 ceph-mon[21811]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T01:06:47.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:46 smithi135 ceph-mon[24080]: pgmap v242: 97 pgs: 97 active+clean; 108 MiB data, 179 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 20 MiB/s wr, 1.10k op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:47.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:46 smithi135 ceph-mon[24080]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T01:06:47.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 4185 22.32 MB/sec warmup 9 sec latency 13.956 ms 2024-04-27T01:06:48.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 5231 22.66 MB/sec warmup 10 sec latency 9.624 ms 2024-04-27T01:06:49.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:48 smithi104 ceph-mon[21811]: pgmap v244: 97 pgs: 97 active+clean; 113 MiB data, 193 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 12 MiB/s wr, 615 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:49.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:48 smithi135 ceph-mon[24080]: pgmap v244: 97 pgs: 97 active+clean; 113 MiB data, 193 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 12 MiB/s wr, 615 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:49.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 6131 22.05 MB/sec warmup 11 sec latency 26.152 ms 2024-04-27T01:06:50.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 7197 24.06 MB/sec warmup 12 sec latency 23.221 ms 2024-04-27T01:06:51.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:50 smithi104 ceph-mon[21811]: pgmap v245: 97 pgs: 97 active+clean; 124 MiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 12 MiB/s wr, 704 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:51.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:50 smithi135 ceph-mon[24080]: pgmap v245: 97 pgs: 97 active+clean; 124 MiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 12 MiB/s wr, 704 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-27T01:06:51.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 7940 22.98 MB/sec warmup 13 sec latency 22.819 ms 2024-04-27T01:06:52.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph mgr module enable nfs --force' 2024-04-27T01:06:52.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 8853 23.04 MB/sec warmup 14 sec latency 18.044 ms 2024-04-27T01:06:53.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:52 smithi104 ceph-mon[21811]: pgmap v246: 97 pgs: 97 active+clean; 136 MiB data, 296 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 10 MiB/s wr, 767 op/s; 174 KiB/s, 0 objects/s recovering 2024-04-27T01:06:53.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:52 smithi135 ceph-mon[24080]: pgmap v246: 97 pgs: 97 active+clean; 136 MiB data, 296 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 10 MiB/s wr, 767 op/s; 174 KiB/s, 0 objects/s recovering 2024-04-27T01:06:53.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 9873 23.09 MB/sec warmup 15 sec latency 17.893 ms 2024-04-27T01:06:54.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 10866 24.27 MB/sec warmup 16 sec latency 22.118 ms 2024-04-27T01:06:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:54 smithi104 ceph-mon[21811]: pgmap v247: 97 pgs: 97 active+clean; 142 MiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 10 MiB/s wr, 855 op/s; 151 KiB/s, 0 objects/s recovering 2024-04-27T01:06:55.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:54 smithi135 ceph-mon[24080]: pgmap v247: 97 pgs: 97 active+clean; 142 MiB data, 320 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 10 MiB/s wr, 855 op/s; 151 KiB/s, 0 objects/s recovering 2024-04-27T01:06:55.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 11488 23.20 MB/sec warmup 17 sec latency 19.087 ms 2024-04-27T01:06:56.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 12432 23.24 MB/sec warmup 18 sec latency 15.869 ms 2024-04-27T01:06:56.970 INFO:teuthology.orchestra.run.smithi104.stderr:module 'nfs' is already enabled 2024-04-27T01:06:57.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:56 smithi104 ceph-mon[21811]: pgmap v248: 97 pgs: 97 active+clean; 156 MiB data, 384 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.12k op/s; 122 KiB/s, 0 objects/s recovering 2024-04-27T01:06:57.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:56 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/488821519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T01:06:57.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:56 smithi135 ceph-mon[24080]: pgmap v248: 97 pgs: 97 active+clean; 156 MiB data, 384 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.12k op/s; 122 KiB/s, 0 objects/s recovering 2024-04-27T01:06:57.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:56 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/488821519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-27T01:06:57.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 13422 23.29 MB/sec warmup 19 sec latency 12.665 ms 2024-04-27T01:06:58.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/488821519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-27T01:06:58.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:57 smithi104 ceph-mon[21811]: mgrmap e21: smithi104.dpemzn(active, since 6m), standbys: smithi135.nwoumo 2024-04-27T01:06:58.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-27T01:06:58.268 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/488821519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-27T01:06:58.268 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:57 smithi135 ceph-mon[24080]: mgrmap e21: smithi104.dpemzn(active, since 6m), standbys: smithi135.nwoumo 2024-04-27T01:06:58.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 14361 24.13 MB/sec warmup 20 sec latency 37.108 ms 2024-04-27T01:06:59.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:06:58 smithi104 ceph-mon[21811]: pgmap v249: 97 pgs: 97 active+clean; 159 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s; 110 KiB/s, 0 objects/s recovering 2024-04-27T01:06:59.286 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:06:58 smithi135 ceph-mon[24080]: pgmap v249: 97 pgs: 97 active+clean; 159 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.08k op/s; 110 KiB/s, 0 objects/s recovering 2024-04-27T01:06:59.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 14961 23.29 MB/sec warmup 21 sec latency 18.543 ms 2024-04-27T01:07:00.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 15932 23.35 MB/sec warmup 22 sec latency 25.818 ms 2024-04-27T01:07:01.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:00 smithi104 ceph-mon[21811]: pgmap v250: 97 pgs: 97 active+clean; 162 MiB data, 455 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 13 MiB/s wr, 1.19k op/s; 102 KiB/s, 0 objects/s recovering 2024-04-27T01:07:01.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:00 smithi135 ceph-mon[24080]: pgmap v250: 97 pgs: 97 active+clean; 162 MiB data, 455 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 13 MiB/s wr, 1.19k op/s; 102 KiB/s, 0 objects/s recovering 2024-04-27T01:07:01.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 16991 23.45 MB/sec warmup 23 sec latency 16.376 ms 2024-04-27T01:07:01.828 INFO:teuthology.orchestra.run.smithi104.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:07:02.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 17902 24.09 MB/sec warmup 24 sec latency 18.165 ms 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: pgmap v251: 97 pgs: 97 active+clean; 172 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 14 MiB/s wr, 1.36k op/s; 102 KiB/s, 0 objects/s recovering 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:02.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:02 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: pgmap v251: 97 pgs: 97 active+clean; 172 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 14 MiB/s wr, 1.36k op/s; 102 KiB/s, 0 objects/s recovering 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:03.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:02 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:03.154 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-27T01:07:03.155 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:07:03.155 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-27T01:07:03.597 INFO:teuthology.orchestra.run.smithi104.stdout: 5 18478 23.37 MB/sec warmup 25 sec latency 23.162 ms 2024-04-27T01:07:03.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:03 smithi104 ceph-mon[21811]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:07:04.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:03 smithi135 ceph-mon[24080]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:07:04.597 INFO:teuthology.orchestra.run.smithi104.stdout: 5 19163 23.17 MB/sec warmup 26 sec latency 42.705 ms 2024-04-27T01:07:04.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:04 smithi104 ceph-mon[21811]: pgmap v252: 97 pgs: 97 active+clean; 169 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-04-27T01:07:05.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:04 smithi135 ceph-mon[24080]: pgmap v252: 97 pgs: 97 active+clean; 169 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-04-27T01:07:05.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 19917 22.69 MB/sec warmup 27 sec latency 46.078 ms 2024-04-27T01:07:06.596 INFO:teuthology.orchestra.run.smithi104.stdout: 5 20421 22.47 MB/sec warmup 28 sec latency 53.261 ms 2024-04-27T01:07:07.150 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:06 smithi104 ceph-mon[21811]: pgmap v253: 97 pgs: 97 active+clean; 175 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-27T01:07:07.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:06 smithi135 ceph-mon[24080]: pgmap v253: 97 pgs: 97 active+clean; 175 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-27T01:07:07.462 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:07:07.597 INFO:teuthology.orchestra.run.smithi104.stdout: 5 21158 22.70 MB/sec warmup 29 sec latency 23.131 ms 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (5m) 31s ago 6m 24.0M - ba2b418f427c a5811d92dc4f 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (6m) 31s ago 6m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (6m) 34s ago 6m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (5m) 31s ago 6m 53.2M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (63s) 31s ago 62s 15.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (60s) 34s ago 60s 14.3M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:07:08.575 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (8m) 31s ago 8m 459M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (6m) 34s ago 6m 415M - 17.2.0 e1d6a67b021e a07c1bcb86e1 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (8m) 31s ago 8m 46.1M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (6m) 34s ago 6m 39.4M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (42s) 31s ago 42s 53.4M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (37s) 34s ago 36s 48.7M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 31s ago 6m 17.2M - 1dbe0e931976 10b041438edc 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 34s ago 5m 19.0M - 1dbe0e931976 10a43cc6931b 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (5m) 31s ago 5m 45.9M 3216M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (5m) 31s ago 5m 42.1M 3216M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (4m) 31s ago 4m 41.0M 3216M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (4m) 31s ago 4m 39.6M 3216M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (3m) 34s ago 3m 39.4M 3984M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (3m) 34s ago 3m 39.1M 3984M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (2m) 34s ago 2m 41.8M 3984M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:07:08.576 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (2m) 34s ago 2m 40.1M 3984M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:07:08.577 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (5m) 31s ago 5m 61.5M - 514e6a882f6e e619d633de42 2024-04-27T01:07:08.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 21666 22.59 MB/sec warmup 30 sec latency 26.224 ms 2024-04-27T01:07:09.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:08 smithi104 ceph-mon[21811]: pgmap v254: 97 pgs: 97 active+clean; 182 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-27T01:07:09.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:08 smithi104 ceph-mon[21811]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:09.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:08 smithi104 ceph-mon[21811]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:07:09.241 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:07:09.242 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:07:09.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:08 smithi135 ceph-mon[24080]: pgmap v254: 97 pgs: 97 active+clean; 182 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-27T01:07:09.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:08 smithi135 ceph-mon[24080]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:09.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:08 smithi135 ceph-mon[24080]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:09.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 22061 21.94 MB/sec warmup 31 sec latency 42.392 ms 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a", 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "", 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a image" 2024-04-27T01:07:09.808 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:07:10.186 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:09 smithi104 ceph-mon[21811]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:10.187 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:09 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3509302601' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:07:10.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:09 smithi135 ceph-mon[24080]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:10.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:09 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3509302601' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:07:10.419 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:07:10.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 22460 21.49 MB/sec warmup 32 sec latency 50.288 ms 2024-04-27T01:07:11.191 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:10 smithi104 ceph-mon[21811]: pgmap v255: 97 pgs: 97 active+clean; 182 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-27T01:07:11.191 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:10 smithi104 ceph-mon[21811]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:11.191 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:10 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4058948759' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:07:11.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:10 smithi135 ceph-mon[24080]: pgmap v255: 97 pgs: 97 active+clean; 182 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-27T01:07:11.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:10 smithi135 ceph-mon[24080]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:11.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:10 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4058948759' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:07:11.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 23205 21.39 MB/sec warmup 33 sec latency 33.106 ms 2024-04-27T01:07:12.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 23890 21.18 MB/sec warmup 34 sec latency 25.236 ms 2024-04-27T01:07:13.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:12 smithi104 ceph-mon[21811]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 998 op/s 2024-04-27T01:07:13.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:12 smithi135 ceph-mon[24080]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 998 op/s 2024-04-27T01:07:13.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 24563 21.43 MB/sec warmup 35 sec latency 21.452 ms 2024-04-27T01:07:14.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 25097 21.41 MB/sec warmup 36 sec latency 45.723 ms 2024-04-27T01:07:15.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:15 smithi104 ceph-mon[21811]: pgmap v257: 97 pgs: 97 active+clean; 189 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 723 op/s 2024-04-27T01:07:15.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:15 smithi135 ceph-mon[24080]: pgmap v257: 97 pgs: 97 active+clean; 189 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 723 op/s 2024-04-27T01:07:15.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 25472 20.95 MB/sec warmup 37 sec latency 39.538 ms 2024-04-27T01:07:16.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 25940 20.58 MB/sec warmup 38 sec latency 21.955 ms 2024-04-27T01:07:17.010 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:17 smithi104 ceph-mon[21811]: pgmap v258: 97 pgs: 97 active+clean; 197 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 925 op/s 2024-04-27T01:07:17.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:17 smithi135 ceph-mon[24080]: pgmap v258: 97 pgs: 97 active+clean; 197 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 925 op/s 2024-04-27T01:07:17.600 INFO:teuthology.orchestra.run.smithi104.stdout: 5 26655 20.54 MB/sec warmup 39 sec latency 43.275 ms 2024-04-27T01:07:18.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 27274 20.23 MB/sec warmup 40 sec latency 40.923 ms 2024-04-27T01:07:19.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:19 smithi104 ceph-mon[21811]: pgmap v259: 97 pgs: 97 active+clean; 193 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 783 op/s 2024-04-27T01:07:19.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:19 smithi135 ceph-mon[24080]: pgmap v259: 97 pgs: 97 active+clean; 193 MiB data, 780 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 783 op/s 2024-04-27T01:07:19.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 28004 20.53 MB/sec warmup 41 sec latency 30.026 ms 2024-04-27T01:07:20.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 28705 20.67 MB/sec warmup 42 sec latency 19.411 ms 2024-04-27T01:07:21.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:21 smithi104 ceph-mon[21811]: pgmap v260: 97 pgs: 97 active+clean; 201 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 772 op/s 2024-04-27T01:07:21.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:21 smithi135 ceph-mon[24080]: pgmap v260: 97 pgs: 97 active+clean; 201 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 772 op/s 2024-04-27T01:07:21.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 29131 20.27 MB/sec warmup 43 sec latency 25.617 ms 2024-04-27T01:07:22.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 29597 20.03 MB/sec warmup 44 sec latency 22.505 ms 2024-04-27T01:07:23.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:23 smithi104 ceph-mon[21811]: pgmap v261: 97 pgs: 97 active+clean; 206 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 853 op/s 2024-04-27T01:07:23.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:23 smithi135 ceph-mon[24080]: pgmap v261: 97 pgs: 97 active+clean; 206 MiB data, 830 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 853 op/s 2024-04-27T01:07:23.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 30301 19.94 MB/sec warmup 45 sec latency 22.113 ms 2024-04-27T01:07:24.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 30853 19.69 MB/sec warmup 46 sec latency 39.648 ms 2024-04-27T01:07:25.147 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:25 smithi104 ceph-mon[21811]: pgmap v262: 97 pgs: 97 active+clean; 207 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 719 op/s 2024-04-27T01:07:25.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:25 smithi135 ceph-mon[24080]: pgmap v262: 97 pgs: 97 active+clean; 207 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 719 op/s 2024-04-27T01:07:25.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 31336 19.75 MB/sec warmup 47 sec latency 36.005 ms 2024-04-27T01:07:26.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:26 smithi104 ceph-mon[21811]: pgmap v263: 97 pgs: 97 active+clean; 215 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 925 op/s 2024-04-27T01:07:26.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:26 smithi135 ceph-mon[24080]: pgmap v263: 97 pgs: 97 active+clean; 215 MiB data, 900 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 925 op/s 2024-04-27T01:07:26.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 31975 19.85 MB/sec warmup 48 sec latency 21.158 ms 2024-04-27T01:07:27.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 32354 19.69 MB/sec warmup 49 sec latency 25.391 ms 2024-04-27T01:07:28.249 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:28 smithi104 ceph-mon[21811]: pgmap v264: 97 pgs: 97 active+clean; 217 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 702 op/s 2024-04-27T01:07:28.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:28 smithi135 ceph-mon[24080]: pgmap v264: 97 pgs: 97 active+clean; 217 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 702 op/s 2024-04-27T01:07:28.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 32775 19.36 MB/sec warmup 50 sec latency 69.453 ms 2024-04-27T01:07:29.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 33181 19.19 MB/sec warmup 51 sec latency 37.827 ms 2024-04-27T01:07:30.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 33972 19.12 MB/sec warmup 52 sec latency 38.531 ms 2024-04-27T01:07:31.030 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:30 smithi104 ceph-mon[21811]: pgmap v265: 97 pgs: 97 active+clean; 222 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 872 op/s 2024-04-27T01:07:31.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:30 smithi135 ceph-mon[24080]: pgmap v265: 97 pgs: 97 active+clean; 222 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 872 op/s 2024-04-27T01:07:31.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 34530 19.00 MB/sec warmup 53 sec latency 36.601 ms 2024-04-27T01:07:32.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 35402 19.33 MB/sec warmup 54 sec latency 21.232 ms 2024-04-27T01:07:33.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:32 smithi104 ceph-mon[21811]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 798 op/s 2024-04-27T01:07:33.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:32 smithi135 ceph-mon[24080]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.4 MiB/s wr, 798 op/s 2024-04-27T01:07:33.601 INFO:teuthology.orchestra.run.smithi104.stdout: 5 35953 19.29 MB/sec warmup 55 sec latency 25.460 ms 2024-04-27T01:07:34.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 36440 19.03 MB/sec warmup 56 sec latency 29.705 ms 2024-04-27T01:07:35.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:34 smithi104 ceph-mon[21811]: pgmap v267: 97 pgs: 97 active+clean; 225 MiB data, 1000 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.6 MiB/s wr, 684 op/s 2024-04-27T01:07:35.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:34 smithi135 ceph-mon[24080]: pgmap v267: 97 pgs: 97 active+clean; 225 MiB data, 1000 MiB used, 714 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.6 MiB/s wr, 684 op/s 2024-04-27T01:07:35.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 37108 19.07 MB/sec warmup 57 sec latency 54.117 ms 2024-04-27T01:07:36.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 37917 18.91 MB/sec warmup 58 sec latency 47.692 ms 2024-04-27T01:07:37.128 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:36 smithi104 ceph-mon[21811]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 899 op/s 2024-04-27T01:07:37.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:36 smithi135 ceph-mon[24080]: pgmap v268: 97 pgs: 97 active+clean; 237 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 899 op/s 2024-04-27T01:07:37.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 38803 19.20 MB/sec warmup 59 sec latency 48.666 ms 2024-04-27T01:07:38.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 39337 19.25 MB/sec warmup 60 sec latency 108.117 ms 2024-04-27T01:07:38.987 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:38 smithi104 ceph-mon[21811]: pgmap v269: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.7 MiB/s wr, 688 op/s 2024-04-27T01:07:39.244 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:38 smithi135 ceph-mon[24080]: pgmap v269: 97 pgs: 97 active+clean; 239 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.7 MiB/s wr, 688 op/s 2024-04-27T01:07:39.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 39916 19.02 MB/sec warmup 61 sec latency 21.023 ms 2024-04-27T01:07:39.996 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:39.996 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T01:07:39.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T01:07:39.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:39.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:39.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:39 smithi104 ceph-mon[21811]: Upgrade: Need to upgrade myself (mgr.smithi104.dpemzn) 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: Upgrade: Target is version 19.0.0-2447-gb22e2ebd (unknown) 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, digests ['quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6'] 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:07:40.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:39 smithi135 ceph-mon[24080]: Upgrade: Need to upgrade myself (mgr.smithi104.dpemzn) 2024-04-27T01:07:40.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 40544 18.98 MB/sec warmup 62 sec latency 21.777 ms 2024-04-27T01:07:40.968 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:07:41.003 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:40 smithi104 ceph-mon[21811]: pgmap v270: 97 pgs: 97 active+clean; 241 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 8.9 MiB/s wr, 791 op/s 2024-04-27T01:07:41.003 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:40 smithi104 ceph-mon[21811]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi135 2024-04-27T01:07:41.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:40 smithi135 ceph-mon[24080]: pgmap v270: 97 pgs: 97 active+clean; 241 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 8.9 MiB/s wr, 791 op/s 2024-04-27T01:07:41.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:40 smithi135 ceph-mon[24080]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 on smithi135 2024-04-27T01:07:41.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 41280 18.86 MB/sec warmup 63 sec latency 35.576 ms 2024-04-27T01:07:41.953 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (5m) 65s ago 7m 24.0M - ba2b418f427c a5811d92dc4f 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (7m) 65s ago 7m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (6m) 67s ago 6m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (5m) 65s ago 7m 53.2M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (97s) 65s ago 96s 15.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (94s) 67s ago 93s 14.3M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (8m) 65s ago 8m 459M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (6m) 67s ago 6m 415M - 17.2.0 e1d6a67b021e a07c1bcb86e1 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (8m) 65s ago 8m 46.1M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (6m) 67s ago 6m 39.4M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (76s) 65s ago 75s 53.4M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (70s) 67s ago 70s 48.7M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:07:41.954 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 65s ago 7m 17.2M - 1dbe0e931976 10b041438edc 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 67s ago 6m 19.0M - 1dbe0e931976 10a43cc6931b 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (6m) 65s ago 6m 45.9M 3216M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (5m) 65s ago 5m 42.1M 3216M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (5m) 65s ago 5m 41.0M 3216M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (4m) 65s ago 4m 39.6M 3216M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (4m) 67s ago 4m 39.4M 3984M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (3m) 67s ago 3m 39.1M 3984M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (3m) 67s ago 3m 41.8M 3984M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (3m) 67s ago 3m 40.1M 3984M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:07:41.955 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (6m) 65s ago 6m 61.5M - 514e6a882f6e e619d633de42 2024-04-27T01:07:42.008 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:41 smithi104 ceph-mon[21811]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:42.008 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:07:42.008 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:41 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:07:42.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:41 smithi135 ceph-mon[24080]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:42.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:07:42.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:41 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:07:42.567 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:07:42.568 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T01:07:42.569 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:07:42.569 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:07:42.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 41947 18.96 MB/sec warmup 64 sec latency 26.634 ms 2024-04-27T01:07:43.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:42 smithi104 ceph-mon[21811]: pgmap v271: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 863 op/s 2024-04-27T01:07:43.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:42 smithi104 ceph-mon[21811]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:43.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:42 smithi104 ceph-mon[21811]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:43.026 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:42 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3056907216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:07:43.042 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 image on host smithi135" 2024-04-27T01:07:43.043 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:07:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:42 smithi135 ceph-mon[24080]: pgmap v271: 97 pgs: 97 active+clean; 251 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 863 op/s 2024-04-27T01:07:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:42 smithi135 ceph-mon[24080]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:42 smithi135 ceph-mon[24080]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:43.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:42 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3056907216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:07:43.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 42701 19.17 MB/sec warmup 65 sec latency 22.559 ms 2024-04-27T01:07:43.668 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:07:44.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:44 smithi104 ceph-mon[21811]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:44.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:44 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/3503590894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:07:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:44 smithi135 ceph-mon[24080]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:44 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/3503590894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:07:44.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 43094 18.98 MB/sec warmup 66 sec latency 34.396 ms 2024-04-27T01:07:45.047 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:45 smithi104 ceph-mon[21811]: pgmap v272: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 785 op/s 2024-04-27T01:07:45.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:45 smithi135 ceph-mon[24080]: pgmap v272: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 785 op/s 2024-04-27T01:07:45.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 43487 18.75 MB/sec warmup 67 sec latency 31.696 ms 2024-04-27T01:07:46.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 43827 18.60 MB/sec warmup 68 sec latency 40.830 ms 2024-04-27T01:07:47.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:47 smithi104 ceph-mon[21811]: pgmap v273: 97 pgs: 97 active+clean; 252 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.7 MiB/s wr, 905 op/s 2024-04-27T01:07:47.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:47 smithi135 ceph-mon[24080]: pgmap v273: 97 pgs: 97 active+clean; 252 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.7 MiB/s wr, 905 op/s 2024-04-27T01:07:47.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 44441 18.55 MB/sec warmup 69 sec latency 49.115 ms 2024-04-27T01:07:48.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 45052 18.44 MB/sec warmup 70 sec latency 53.721 ms 2024-04-27T01:07:49.196 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:49 smithi104 ceph-mon[21811]: pgmap v274: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 771 op/s 2024-04-27T01:07:49.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:49 smithi135 ceph-mon[24080]: pgmap v274: 97 pgs: 97 active+clean; 256 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 771 op/s 2024-04-27T01:07:49.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 45472 18.41 MB/sec warmup 71 sec latency 45.217 ms 2024-04-27T01:07:50.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 46109 18.54 MB/sec warmup 72 sec latency 29.290 ms 2024-04-27T01:07:51.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:51 smithi104 ceph-mon[21811]: pgmap v275: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 906 op/s 2024-04-27T01:07:51.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:51 smithi135 ceph-mon[24080]: pgmap v275: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 906 op/s 2024-04-27T01:07:51.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 46517 18.45 MB/sec warmup 73 sec latency 44.446 ms 2024-04-27T01:07:52.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 46957 18.26 MB/sec warmup 74 sec latency 44.994 ms 2024-04-27T01:07:53.018 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:53 smithi104 ceph-mon[21811]: pgmap v276: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 969 op/s 2024-04-27T01:07:53.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:53 smithi135 ceph-mon[24080]: pgmap v276: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.2 MiB/s wr, 969 op/s 2024-04-27T01:07:53.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 47578 18.23 MB/sec warmup 75 sec latency 41.642 ms 2024-04-27T01:07:54.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 48239 18.12 MB/sec warmup 76 sec latency 39.045 ms 2024-04-27T01:07:55.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:55 smithi104 ceph-mon[21811]: pgmap v277: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 734 op/s 2024-04-27T01:07:55.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:55 smithi135 ceph-mon[24080]: pgmap v277: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 734 op/s 2024-04-27T01:07:55.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 48649 18.02 MB/sec warmup 77 sec latency 80.682 ms 2024-04-27T01:07:56.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 49333 18.16 MB/sec warmup 78 sec latency 32.627 ms 2024-04-27T01:07:57.058 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:57 smithi104 ceph-mon[21811]: pgmap v278: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.7 MiB/s wr, 753 op/s 2024-04-27T01:07:57.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:57 smithi135 ceph-mon[24080]: pgmap v278: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.7 MiB/s wr, 753 op/s 2024-04-27T01:07:57.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 49972 18.25 MB/sec warmup 79 sec latency 32.904 ms 2024-04-27T01:07:58.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 50537 18.09 MB/sec warmup 80 sec latency 23.612 ms 2024-04-27T01:07:59.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:07:59 smithi104 ceph-mon[21811]: pgmap v279: 97 pgs: 97 active+clean; 272 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.2 MiB/s wr, 736 op/s 2024-04-27T01:07:59.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:07:59 smithi135 ceph-mon[24080]: pgmap v279: 97 pgs: 97 active+clean; 272 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.2 MiB/s wr, 736 op/s 2024-04-27T01:07:59.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 51162 18.07 MB/sec warmup 81 sec latency 38.114 ms 2024-04-27T01:08:00.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 51913 17.98 MB/sec warmup 82 sec latency 32.662 ms 2024-04-27T01:08:01.131 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:01 smithi104 ceph-mon[21811]: pgmap v280: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.1 MiB/s wr, 763 op/s 2024-04-27T01:08:01.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:01 smithi135 ceph-mon[24080]: pgmap v280: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.1 MiB/s wr, 763 op/s 2024-04-27T01:08:01.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 52390 17.93 MB/sec warmup 83 sec latency 27.458 ms 2024-04-27T01:08:02.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 53123 18.09 MB/sec warmup 84 sec latency 40.648 ms 2024-04-27T01:08:03.159 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:03 smithi104 ceph-mon[21811]: pgmap v281: 97 pgs: 97 active+clean; 289 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 811 op/s 2024-04-27T01:08:03.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:03 smithi135 ceph-mon[24080]: pgmap v281: 97 pgs: 97 active+clean; 289 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 811 op/s 2024-04-27T01:08:03.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 53604 18.09 MB/sec warmup 85 sec latency 62.889 ms 2024-04-27T01:08:04.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 54106 17.94 MB/sec warmup 86 sec latency 27.207 ms 2024-04-27T01:08:05.175 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:05 smithi104 ceph-mon[21811]: pgmap v282: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.6 MiB/s wr, 653 op/s 2024-04-27T01:08:05.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:05 smithi135 ceph-mon[24080]: pgmap v282: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.6 MiB/s wr, 653 op/s 2024-04-27T01:08:05.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 54622 17.88 MB/sec warmup 87 sec latency 29.637 ms 2024-04-27T01:08:06.603 INFO:teuthology.orchestra.run.smithi104.stdout: 5 55175 17.81 MB/sec warmup 88 sec latency 40.430 ms 2024-04-27T01:08:07.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:07 smithi104 ceph-mon[21811]: pgmap v283: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 772 op/s 2024-04-27T01:08:07.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:07 smithi135 ceph-mon[24080]: pgmap v283: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 772 op/s 2024-04-27T01:08:07.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 55747 17.75 MB/sec warmup 89 sec latency 30.140 ms 2024-04-27T01:08:08.602 INFO:teuthology.orchestra.run.smithi104.stdout: 5 56197 17.78 MB/sec warmup 90 sec latency 31.997 ms 2024-04-27T01:08:09.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:09 smithi104 ceph-mon[21811]: pgmap v284: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.5 MiB/s wr, 821 op/s 2024-04-27T01:08:09.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:09 smithi135 ceph-mon[24080]: pgmap v284: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.5 MiB/s wr, 821 op/s 2024-04-27T01:08:09.608 INFO:teuthology.orchestra.run.smithi104.stdout: 5 56935 17.90 MB/sec warmup 91 sec latency 29.368 ms 2024-04-27T01:08:10.608 INFO:teuthology.orchestra.run.smithi104.stdout: 5 57198 17.76 MB/sec warmup 92 sec latency 54.076 ms 2024-04-27T01:08:11.124 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:11 smithi104 ceph-mon[21811]: pgmap v285: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 9.3 MiB/s wr, 916 op/s 2024-04-27T01:08:11.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:11 smithi135 ceph-mon[24080]: pgmap v285: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 KiB/s rd, 9.3 MiB/s wr, 916 op/s 2024-04-27T01:08:11.608 INFO:teuthology.orchestra.run.smithi104.stdout: 5 57494 17.60 MB/sec warmup 93 sec latency 51.245 ms 2024-04-27T01:08:12.608 INFO:teuthology.orchestra.run.smithi104.stdout: 5 57795 17.47 MB/sec warmup 94 sec latency 67.997 ms 2024-04-27T01:08:13.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:13 smithi104 ceph-mon[21811]: pgmap v286: 97 pgs: 97 active+clean; 302 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 896 op/s 2024-04-27T01:08:13.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:13 smithi135 ceph-mon[24080]: pgmap v286: 97 pgs: 97 active+clean; 302 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 896 op/s 2024-04-27T01:08:13.608 INFO:teuthology.orchestra.run.smithi104.stdout: 5 58300 17.42 MB/sec warmup 95 sec latency 43.152 ms 2024-04-27T01:08:14.199 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:08:14.224 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:14 smithi104 ceph-mon[21811]: pgmap v287: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 708 op/s 2024-04-27T01:08:14.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:14 smithi135 ceph-mon[24080]: pgmap v287: 97 pgs: 97 active+clean; 307 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 708 op/s 2024-04-27T01:08:14.611 INFO:teuthology.orchestra.run.smithi104.stdout: 5 58990 17.35 MB/sec warmup 96 sec latency 25.374 ms 2024-04-27T01:08:15.196 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:08:15.196 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (6m) 98s ago 8m 24.0M - ba2b418f427c a5811d92dc4f 2024-04-27T01:08:15.196 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (7m) 98s ago 7m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:08:15.196 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (7m) 100s ago 7m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:08:15.196 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (6m) 98s ago 7m 53.2M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (2m) 98s ago 2m 15.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (2m) 100s ago 2m 14.3M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (9m) 98s ago 9m 459M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (7m) 100s ago 7m 415M - 17.2.0 e1d6a67b021e a07c1bcb86e1 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (9m) 98s ago 9m 46.1M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (7m) 100s ago 7m 39.4M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (109s) 98s ago 108s 53.4M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (104s) 100s ago 103s 48.7M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 98s ago 7m 17.2M - 1dbe0e931976 10b041438edc 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 100s ago 7m 19.0M - 1dbe0e931976 10a43cc6931b 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (6m) 98s ago 6m 45.9M 3216M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (6m) 98s ago 6m 42.1M 3216M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (5m) 98s ago 5m 41.0M 3216M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:08:15.197 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (5m) 98s ago 5m 39.6M 3216M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:08:15.198 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (4m) 100s ago 4m 39.4M 3984M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:08:15.198 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (4m) 100s ago 4m 39.1M 3984M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:08:15.198 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (4m) 100s ago 4m 41.8M 3984M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:08:15.198 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (3m) 100s ago 3m 40.1M 3984M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:08:15.198 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (6m) 98s ago 6m 61.5M - 514e6a882f6e e619d633de42 2024-04-27T01:08:15.613 INFO:teuthology.orchestra.run.smithi104.stdout: 5 59712 17.46 MB/sec warmup 97 sec latency 22.258 ms 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:15.775 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:08:15.776 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: Upgrade: Updating mgr.smithi135.nwoumo 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:08:15.984 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:15.985 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:08:15.985 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:15 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/300680839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: Upgrade: Updating mgr.smithi135.nwoumo 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:08:16.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:08:16.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:16.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:08:16.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:15 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/300680839' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "0/23 daemons upgraded", 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading mgr daemons" 2024-04-27T01:08:16.221 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:08:16.613 INFO:teuthology.orchestra.run.smithi104.stdout: 5 60313 17.52 MB/sec warmup 98 sec latency 123.541 ms 2024-04-27T01:08:16.731 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:08:17.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:16 smithi104 ceph-mon[21811]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-27T01:08:17.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:16 smithi104 ceph-mon[21811]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:17.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:16 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/607134380' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:08:17.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:16 smithi135 ceph-mon[24080]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-27T01:08:17.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:16 smithi135 ceph-mon[24080]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:17.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:16 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/607134380' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:08:17.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 60725 17.47 MB/sec warmup 99 sec latency 35.246 ms 2024-04-27T01:08:18.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:17 smithi104 ceph-mon[21811]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:18.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:17 smithi135 ceph-mon[24080]: from='client.14678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:18.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 60944 17.32 MB/sec warmup 100 sec latency 145.964 ms 2024-04-27T01:08:19.201 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:18 smithi104 ceph-mon[21811]: pgmap v289: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.3 MiB/s wr, 795 op/s 2024-04-27T01:08:19.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:18 smithi135 ceph-mon[24080]: pgmap v289: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.3 MiB/s wr, 795 op/s 2024-04-27T01:08:19.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 61309 17.20 MB/sec warmup 101 sec latency 143.738 ms 2024-04-27T01:08:20.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 61702 17.10 MB/sec warmup 102 sec latency 60.345 ms 2024-04-27T01:08:21.230 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:20 smithi104 ceph-mon[21811]: pgmap v290: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 746 op/s 2024-04-27T01:08:21.278 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:20 smithi135 ceph-mon[24080]: pgmap v290: 97 pgs: 97 active+clean; 314 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 746 op/s 2024-04-27T01:08:21.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 62233 17.07 MB/sec warmup 103 sec latency 51.082 ms 2024-04-27T01:08:22.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 62917 17.04 MB/sec warmup 104 sec latency 40.895 ms 2024-04-27T01:08:23.013 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:22 smithi104 ceph-mon[21811]: pgmap v291: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 670 op/s 2024-04-27T01:08:23.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:22 smithi135 ceph-mon[24080]: pgmap v291: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 670 op/s 2024-04-27T01:08:23.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 63570 17.16 MB/sec warmup 105 sec latency 65.266 ms 2024-04-27T01:08:24.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 64064 17.18 MB/sec warmup 106 sec latency 91.008 ms 2024-04-27T01:08:25.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:24 smithi104 ceph-mon[21811]: pgmap v292: 97 pgs: 97 active+clean; 327 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 657 op/s 2024-04-27T01:08:25.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:24 smithi135 ceph-mon[24080]: pgmap v292: 97 pgs: 97 active+clean; 327 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.7 MiB/s wr, 657 op/s 2024-04-27T01:08:25.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 64430 17.08 MB/sec warmup 107 sec latency 30.880 ms 2024-04-27T01:08:26.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 64837 16.97 MB/sec warmup 108 sec latency 107.880 ms 2024-04-27T01:08:26.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:26 smithi104 ceph-mon[21811]: pgmap v293: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 683 op/s 2024-04-27T01:08:26.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:26.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:26.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:26.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:26 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:27.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:26 smithi135 ceph-mon[24080]: pgmap v293: 97 pgs: 97 active+clean; 325 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 683 op/s 2024-04-27T01:08:27.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:27.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:27.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:27.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:26 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:27.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 65341 16.91 MB/sec warmup 109 sec latency 32.285 ms 2024-04-27T01:08:28.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 65941 16.87 MB/sec warmup 110 sec latency 39.349 ms 2024-04-27T01:08:28.861 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:28 smithi104 ceph-mon[21811]: pgmap v294: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 7.1 MiB/s wr, 721 op/s 2024-04-27T01:08:29.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:28 smithi135 ceph-mon[24080]: pgmap v294: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 7.1 MiB/s wr, 721 op/s 2024-04-27T01:08:29.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 66441 16.82 MB/sec warmup 111 sec latency 47.600 ms 2024-04-27T01:08:30.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 66883 16.85 MB/sec warmup 112 sec latency 34.308 ms 2024-04-27T01:08:31.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:30 smithi104 ceph-mon[21811]: pgmap v295: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 7.3 MiB/s wr, 651 op/s 2024-04-27T01:08:31.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:30 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:31.226 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:30 smithi135 ceph-mon[24080]: pgmap v295: 97 pgs: 97 active+clean; 331 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 7.3 MiB/s wr, 651 op/s 2024-04-27T01:08:31.227 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:30 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:31.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 67313 16.85 MB/sec warmup 113 sec latency 50.873 ms 2024-04-27T01:08:32.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:31 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:32.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:31 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:32.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 67744 16.84 MB/sec warmup 114 sec latency 39.389 ms 2024-04-27T01:08:33.005 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:32 smithi135 ceph-mon[24080]: pgmap v296: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 796 op/s 2024-04-27T01:08:33.076 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:32 smithi104 ceph-mon[21811]: pgmap v296: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 796 op/s 2024-04-27T01:08:33.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 68024 16.72 MB/sec warmup 115 sec latency 44.128 ms 2024-04-27T01:08:34.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 68569 16.65 MB/sec warmup 116 sec latency 34.160 ms 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: pgmap v297: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.4 MiB/s wr, 667 op/s 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:08:35.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr fail", "who": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:08:35.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:34 smithi104 ceph-mon[21811]: osdmap e74: 8 total, 8 up, 8 in 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: pgmap v297: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.4 MiB/s wr, 667 op/s 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:35.242 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr fail", "who": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:08:35.243 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:34 smithi135 ceph-mon[24080]: osdmap e74: 8 total, 8 up, 8 in 2024-04-27T01:08:35.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 69119 16.63 MB/sec warmup 117 sec latency 44.499 ms 2024-04-27T01:08:36.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:08:36.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "mgr fail", "who": "smithi104.dpemzn"}]': finished 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: mgrmap e22: smithi135.nwoumo(active, starting, since 0.420701s) 2024-04-27T01:08:36.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: Manager daemon smithi135.nwoumo is now available 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/trash_purge_schedule"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/trash_purge_schedule"}]: dispatch 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.111 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:36 smithi104 ceph-mon[21811]: Deploying cephadm binary to smithi135 2024-04-27T01:08:36.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:08:36.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.14202 172.21.15.104:0/3939072079' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "mgr fail", "who": "smithi104.dpemzn"}]': finished 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: mgrmap e22: smithi135.nwoumo(active, starting, since 0.420701s) 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: Manager daemon smithi135.nwoumo is now available 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/trash_purge_schedule"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi135.nwoumo/trash_purge_schedule"}]: dispatch 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.368 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:36.369 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:36 smithi135 ceph-mon[24080]: Deploying cephadm binary to smithi135 2024-04-27T01:08:36.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 69785 16.60 MB/sec warmup 118 sec latency 29.962 ms 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: Deploying cephadm binary to smithi104 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: mgrmap e23: smithi135.nwoumo(active, since 1.4225s) 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: [27/Apr/2024:01:08:36] ENGINE Bus STARTING 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: [27/Apr/2024:01:08:36] ENGINE Serving on https://172.21.15.135:7150 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: [27/Apr/2024:01:08:36] ENGINE Serving on http://172.21.15.135:8765 2024-04-27T01:08:37.116 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:37 smithi104 ceph-mon[21811]: [27/Apr/2024:01:08:36] ENGINE Bus STARTED 2024-04-27T01:08:37.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: Deploying cephadm binary to smithi104 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: mgrmap e23: smithi135.nwoumo(active, since 1.4225s) 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: [27/Apr/2024:01:08:36] ENGINE Bus STARTING 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: [27/Apr/2024:01:08:36] ENGINE Serving on https://172.21.15.135:7150 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: [27/Apr/2024:01:08:36] ENGINE Serving on http://172.21.15.135:8765 2024-04-27T01:08:37.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:37 smithi135 ceph-mon[24080]: [27/Apr/2024:01:08:36] ENGINE Bus STARTED 2024-04-27T01:08:37.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 70335 16.63 MB/sec warmup 119 sec latency 30.700 ms 2024-04-27T01:08:38.121 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:38 smithi104 ceph-mon[21811]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:38.122 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:38 smithi104 ceph-mon[21811]: mgrmap e24: smithi135.nwoumo(active, since 2s) 2024-04-27T01:08:38.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:38 smithi135 ceph-mon[24080]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:38.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:38 smithi135 ceph-mon[24080]: mgrmap e24: smithi135.nwoumo(active, since 2s) 2024-04-27T01:08:39.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 71277 12.43 MB/sec execute 1 sec latency 60.718 ms 2024-04-27T01:08:40.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:40 smithi104 ceph-mon[21811]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:40.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:40 smithi104 ceph-mon[21811]: mgrmap e25: smithi135.nwoumo(active, since 4s) 2024-04-27T01:08:40.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:40 smithi135 ceph-mon[24080]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:08:40.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:40 smithi135 ceph-mon[24080]: mgrmap e25: smithi135.nwoumo(active, since 4s) 2024-04-27T01:08:40.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 71536 8.54 MB/sec execute 2 sec latency 68.413 ms 2024-04-27T01:08:41.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 71937 7.54 MB/sec execute 3 sec latency 47.567 ms 2024-04-27T01:08:42.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:42 smithi104 ceph-mon[21811]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s wr, 62 op/s 2024-04-27T01:08:42.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:42 smithi135 ceph-mon[24080]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s wr, 62 op/s 2024-04-27T01:08:42.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 72382 7.97 MB/sec execute 4 sec latency 29.556 ms 2024-04-27T01:08:43.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 72895 8.98 MB/sec execute 5 sec latency 59.300 ms 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: pgmap v7: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 5.6 MiB/s wr, 491 op/s 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/crt"}]: dispatch 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/key"}]: dispatch 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:08:44.246 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:44 smithi104 ceph-mon[21811]: Standby manager daemon smithi104.dpemzn started 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: pgmap v7: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 5.6 MiB/s wr, 491 op/s 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/crt"}]: dispatch 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/key"}]: dispatch 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/1771109864' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:08:44.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:44 smithi135 ceph-mon[24080]: Standby manager daemon smithi104.dpemzn started 2024-04-27T01:08:44.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 73647 10.38 MB/sec execute 6 sec latency 24.505 ms 2024-04-27T01:08:45.250 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:45 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:08:45.251 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:45 smithi104 ceph-mon[21811]: mgrmap e26: smithi135.nwoumo(active, since 9s), standbys: smithi104.dpemzn 2024-04-27T01:08:45.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:45 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:08:45.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:45 smithi135 ceph-mon[24080]: mgrmap e26: smithi135.nwoumo(active, since 9s), standbys: smithi104.dpemzn 2024-04-27T01:08:45.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 74298 12.60 MB/sec execute 7 sec latency 25.722 ms 2024-04-27T01:08:46.261 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:46 smithi104 ceph-mon[21811]: pgmap v8: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.3 MiB/s wr, 381 op/s 2024-04-27T01:08:46.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:46 smithi135 ceph-mon[24080]: pgmap v8: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.3 MiB/s wr, 381 op/s 2024-04-27T01:08:46.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 74787 13.15 MB/sec execute 8 sec latency 35.095 ms 2024-04-27T01:08:47.268 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:08:47.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 75071 12.55 MB/sec execute 9 sec latency 48.644 ms 2024-04-27T01:08:48.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:48 smithi104 ceph-mon[21811]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-04-27T01:08:48.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:48 smithi104 ceph-mon[21811]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (7m) 2m ago 8m 24.0M - ba2b418f427c a5811d92dc4f 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (7m) 18s ago 7m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (6m) 2m ago 8m 53.2M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (2m) 2m ago 2m 15.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (2m) 18s ago 2m 14.8M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (9m) 2m ago 9m 459M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:08:48.191 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (23s) 18s ago 7m 43.5M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 cd90a4a984a6 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (9m) 2m ago 9m 46.1M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (7m) 18s ago 7m 36.9M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (2m) 2m ago 2m 53.4M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (2m) 18s ago 2m 73.0M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 2m ago 8m 17.2M - 1dbe0e931976 10b041438edc 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 18s ago 7m 18.6M - 1dbe0e931976 10a43cc6931b 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (7m) 2m ago 7m 45.9M 3216M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (6m) 2m ago 6m 42.1M 3216M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (6m) 2m ago 6m 41.0M 3216M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (5m) 2m ago 5m 39.6M 3216M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:08:48.193 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (5m) 18s ago 5m 245M 3984M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:08:48.194 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (4m) 18s ago 4m 279M 3984M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:08:48.194 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (4m) 18s ago 4m 250M 3984M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:08:48.194 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (4m) 18s ago 4m 333M 3984M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:08:48.194 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (7m) 2m ago 7m 61.5M - 514e6a882f6e e619d633de42 2024-04-27T01:08:48.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:48 smithi135 ceph-mon[24080]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 7.2 MiB/s wr, 664 op/s 2024-04-27T01:08:48.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:48 smithi135 ceph-mon[24080]: from='client.24447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:48.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 75446 11.81 MB/sec execute 10 sec latency 73.330 ms 2024-04-27T01:08:48.845 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:08:48.846 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:08:48.847 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:08:49.131 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:49 smithi104 ceph-mon[21811]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:49.131 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:49 smithi104 ceph-mon[21811]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:49.131 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:49 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1723526325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:08:49.338 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:08:49.338 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:08:49.338 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:08:49.338 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:08:49.338 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:08:49.339 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T01:08:49.339 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:08:49.339 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:08:49.339 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:08:49.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:49 smithi135 ceph-mon[24080]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:49.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:49 smithi135 ceph-mon[24080]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:49.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:49 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1723526325' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:08:49.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 75884 11.31 MB/sec execute 11 sec latency 35.598 ms 2024-04-27T01:08:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:08:50.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-04-27T01:08:50.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:50.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.604 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.605 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.605 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.605 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2657580902' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:08:50.605 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:50 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:08:50.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 76371 11.46 MB/sec execute 12 sec latency 39.933 ms 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2657580902' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:08:50.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:50 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:08:51.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 77023 11.60 MB/sec execute 13 sec latency 29.974 ms 2024-04-27T01:08:51.930 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: pgmap v11: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:51.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi104 to 1936M 2024-04-27T01:08:51.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:51 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: pgmap v11: 97 pgs: 97 active+clean; 324 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi104 to 1936M 2024-04-27T01:08:52.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:51 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:52.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 77486 12.21 MB/sec execute 14 sec latency 28.057 ms 2024-04-27T01:08:53.615 INFO:teuthology.orchestra.run.smithi104.stdout: 5 78099 12.88 MB/sec execute 15 sec latency 40.201 ms 2024-04-27T01:08:54.250 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:54 smithi104 ceph-mon[21811]: pgmap v12: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 846 op/s 2024-04-27T01:08:54.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:54 smithi135 ceph-mon[24080]: pgmap v12: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 846 op/s 2024-04-27T01:08:54.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 78535 12.94 MB/sec execute 16 sec latency 31.850 ms 2024-04-27T01:08:55.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 78894 12.53 MB/sec execute 17 sec latency 31.143 ms 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.760 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Adjusting osd_memory_target on smithi135 to 2704M 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 586 op/s 2024-04-27T01:08:55.761 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:08:55.762 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:08:55.762 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:08:55.762 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:55 smithi104 ceph-mon[21811]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:08:56.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:56.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:56.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Adjusting osd_memory_target on smithi135 to 2704M 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:56.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.6 MiB/s wr, 586 op/s 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:08:56.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:55 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:08:56.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 79279 12.09 MB/sec execute 18 sec latency 49.713 ms 2024-04-27T01:08:57.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.034 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: Reconfiguring alertmanager.smithi104 (dependencies changed)... 2024-04-27T01:08:57.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:56 smithi104 ceph-mon[21811]: Reconfiguring daemon alertmanager.smithi104 on smithi104 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:08:57.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: Reconfiguring alertmanager.smithi104 (dependencies changed)... 2024-04-27T01:08:57.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:56 smithi135 ceph-mon[24080]: Reconfiguring daemon alertmanager.smithi104 on smithi104 2024-04-27T01:08:57.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 79739 12.07 MB/sec execute 19 sec latency 39.659 ms 2024-04-27T01:08:57.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:08:57 smithi104 ceph-mon[21811]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 874 op/s 2024-04-27T01:08:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:08:57 smithi135 ceph-mon[24080]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 874 op/s 2024-04-27T01:08:58.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 80340 12.20 MB/sec execute 20 sec latency 43.649 ms 2024-04-27T01:08:59.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 80875 12.30 MB/sec execute 21 sec latency 35.710 ms 2024-04-27T01:09:00.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:00 smithi104 ceph-mon[21811]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 552 op/s 2024-04-27T01:09:00.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:00 smithi135 ceph-mon[24080]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 552 op/s 2024-04-27T01:09:00.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 81353 12.67 MB/sec execute 22 sec latency 31.365 ms 2024-04-27T01:09:01.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 81809 12.86 MB/sec execute 23 sec latency 40.767 ms 2024-04-27T01:09:02.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:02 smithi104 ceph-mon[21811]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.9 MiB/s wr, 690 op/s 2024-04-27T01:09:02.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:02 smithi135 ceph-mon[24080]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.9 MiB/s wr, 690 op/s 2024-04-27T01:09:02.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 82228 12.67 MB/sec execute 24 sec latency 57.001 ms 2024-04-27T01:09:03.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 82737 12.48 MB/sec execute 25 sec latency 78.537 ms 2024-04-27T01:09:04.172 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:04 smithi104 ceph-mon[21811]: pgmap v17: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-04-27T01:09:04.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:04 smithi135 ceph-mon[24080]: pgmap v17: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-04-27T01:09:04.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 83331 12.60 MB/sec execute 26 sec latency 29.270 ms 2024-04-27T01:09:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 84033 12.63 MB/sec execute 27 sec latency 33.895 ms 2024-04-27T01:09:06.191 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:05 smithi104 ceph-mon[21811]: pgmap v18: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-04-27T01:09:06.192 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:06.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:05 smithi135 ceph-mon[24080]: pgmap v18: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-04-27T01:09:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:06.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 84526 12.85 MB/sec execute 28 sec latency 32.794 ms 2024-04-27T01:09:07.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 85075 13.18 MB/sec execute 29 sec latency 43.862 ms 2024-04-27T01:09:08.208 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:08 smithi104 ceph-mon[21811]: pgmap v19: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 907 op/s 2024-04-27T01:09:08.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:08 smithi135 ceph-mon[24080]: pgmap v19: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 907 op/s 2024-04-27T01:09:08.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 85536 13.18 MB/sec execute 30 sec latency 70.365 ms 2024-04-27T01:09:09.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 85943 13.03 MB/sec execute 31 sec latency 32.827 ms 2024-04-27T01:09:10.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:10 smithi104 ceph-mon[21811]: pgmap v20: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-04-27T01:09:10.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:10 smithi135 ceph-mon[24080]: pgmap v20: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.1 MiB/s wr, 619 op/s 2024-04-27T01:09:10.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 86457 12.80 MB/sec execute 32 sec latency 44.284 ms 2024-04-27T01:09:11.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 86914 12.84 MB/sec execute 33 sec latency 24.520 ms 2024-04-27T01:09:12.244 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:12 smithi104 ceph-mon[21811]: pgmap v21: 97 pgs: 97 active+clean; 324 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.3 MiB/s wr, 729 op/s 2024-04-27T01:09:12.244 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:12 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:12.245 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:12 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:12.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:12 smithi135 ceph-mon[24080]: pgmap v21: 97 pgs: 97 active+clean; 324 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.3 MiB/s wr, 729 op/s 2024-04-27T01:09:12.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:12 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:12.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:12 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:12.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 87433 12.80 MB/sec execute 34 sec latency 31.046 ms 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: Reconfiguring grafana.smithi104 (dependencies changed)... 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:09:13.243 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:13 smithi104 ceph-mon[21811]: Reconfiguring daemon grafana.smithi104 on smithi104 2024-04-27T01:09:13.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: Reconfiguring grafana.smithi104 (dependencies changed)... 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:09:13.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:13 smithi135 ceph-mon[24080]: Reconfiguring daemon grafana.smithi104 on smithi104 2024-04-27T01:09:13.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 87954 12.91 MB/sec execute 35 sec latency 26.984 ms 2024-04-27T01:09:14.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:14 smithi104 ceph-mon[21811]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 773 op/s 2024-04-27T01:09:14.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:14 smithi135 ceph-mon[24080]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 773 op/s 2024-04-27T01:09:14.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 88423 13.03 MB/sec execute 36 sec latency 32.656 ms 2024-04-27T01:09:15.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 88884 13.16 MB/sec execute 37 sec latency 37.946 ms 2024-04-27T01:09:16.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:15 smithi104 ceph-mon[21811]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 618 op/s 2024-04-27T01:09:16.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:15 smithi135 ceph-mon[24080]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 618 op/s 2024-04-27T01:09:16.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 89219 13.01 MB/sec execute 38 sec latency 93.294 ms 2024-04-27T01:09:17.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 89640 12.83 MB/sec execute 39 sec latency 39.358 ms 2024-04-27T01:09:18.130 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:18 smithi104 ceph-mon[21811]: pgmap v24: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 895 op/s 2024-04-27T01:09:18.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:18 smithi135 ceph-mon[24080]: pgmap v24: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 895 op/s 2024-04-27T01:09:18.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 90151 12.82 MB/sec execute 40 sec latency 60.450 ms 2024-04-27T01:09:19.616 INFO:teuthology.orchestra.run.smithi104.stdout: 5 90605 12.75 MB/sec execute 41 sec latency 44.207 ms 2024-04-27T01:09:20.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:20 smithi104 ceph-mon[21811]: pgmap v25: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 568 op/s 2024-04-27T01:09:20.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:20 smithi135 ceph-mon[24080]: pgmap v25: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 568 op/s 2024-04-27T01:09:20.550 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:09:20.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 91152 12.68 MB/sec execute 42 sec latency 31.598 ms 2024-04-27T01:09:21.192 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:21 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:21.192 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:21 smithi104 ceph-mon[21811]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:21.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:21 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:21.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:21 smithi135 ceph-mon[24080]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:21.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 91602 12.80 MB/sec execute 43 sec latency 33.191 ms 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 starting - - - - 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (9m) 31s ago 9m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (8m) 32s ago 8m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (7m) 31s ago 8m 52.8M - 8.3.5 dad864ee21e9 432482fd33c8 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (3m) 31s ago 3m 38.9M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (3m) 32s ago 3m 15.1M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (10m) 31s ago 10m 349M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (56s) 32s ago 8m 488M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 cd90a4a984a6 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (10m) 31s ago 10m 45.8M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (8m) 32s ago 8m 40.0M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (2m) 31s ago 2m 265M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:09:21.713 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (2m) 32s ago 2m 73.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 31s ago 8m 21.6M - 1.3.1 1dbe0e931976 10b041438edc 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 32s ago 8m 21.2M - 1.3.1 1dbe0e931976 10a43cc6931b 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (7m) 31s ago 7m 293M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (7m) 31s ago 7m 234M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (6m) 31s ago 6m 208M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (6m) 31s ago 6m 255M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (5m) 32s ago 5m 266M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (5m) 32s ago 5m 299M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (5m) 32s ago 5m 268M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (4m) 32s ago 4m 353M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:09:21.714 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (7m) 31s ago 7m 65.8M - 2.33.4 514e6a882f6e e619d633de42 2024-04-27T01:09:22.198 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:22 smithi104 ceph-mon[21811]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 638 op/s 2024-04-27T01:09:22.198 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:22 smithi104 ceph-mon[21811]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:22 smithi135 ceph-mon[24080]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 638 op/s 2024-04-27T01:09:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:22 smithi135 ceph-mon[24080]: from='client.24473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:22.429 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:09:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:22.431 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:09:22.431 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T01:09:22.431 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:09:22.431 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:09:22.431 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:09:22.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 92159 13.04 MB/sec execute 44 sec latency 32.451 ms 2024-04-27T01:09:22.973 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:09:22.974 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:09:22.974 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:09:22.974 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:09:22.974 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:09:22.974 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T01:09:22.975 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:09:22.975 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:09:22.975 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:09:23.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:23 smithi104 ceph-mon[21811]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:23.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:23 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1341650972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:09:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:23 smithi135 ceph-mon[24080]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:23.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:23 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1341650972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:09:23.532 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:09:23.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 92568 13.03 MB/sec execute 45 sec latency 42.043 ms 2024-04-27T01:09:24.233 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:24 smithi104 ceph-mon[21811]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:24.233 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:24 smithi104 ceph-mon[21811]: pgmap v27: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 736 op/s 2024-04-27T01:09:24.233 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:24 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2962842359' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:09:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:24 smithi135 ceph-mon[24080]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:24 smithi135 ceph-mon[24080]: pgmap v27: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.6 MiB/s wr, 736 op/s 2024-04-27T01:09:24.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:24 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2962842359' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:09:24.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 92912 12.86 MB/sec execute 46 sec latency 33.294 ms 2024-04-27T01:09:25.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 93305 12.70 MB/sec execute 47 sec latency 44.945 ms 2024-04-27T01:09:26.006 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:25 smithi104 ceph-mon[21811]: pgmap v28: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 556 op/s 2024-04-27T01:09:26.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:25 smithi135 ceph-mon[24080]: pgmap v28: 97 pgs: 97 active+clean; 317 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 556 op/s 2024-04-27T01:09:26.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 93726 12.70 MB/sec execute 48 sec latency 41.075 ms 2024-04-27T01:09:27.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 94236 12.63 MB/sec execute 49 sec latency 46.091 ms 2024-04-27T01:09:28.293 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:28 smithi104 ceph-mon[21811]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 819 op/s 2024-04-27T01:09:28.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:28 smithi135 ceph-mon[24080]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 819 op/s 2024-04-27T01:09:28.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 94794 12.60 MB/sec execute 50 sec latency 62.169 ms 2024-04-27T01:09:29.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 95219 12.72 MB/sec execute 51 sec latency 29.595 ms 2024-04-27T01:09:30.323 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:30 smithi104 ceph-mon[21811]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 541 op/s 2024-04-27T01:09:30.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:30 smithi135 ceph-mon[24080]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 541 op/s 2024-04-27T01:09:30.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 95809 12.91 MB/sec execute 52 sec latency 38.356 ms 2024-04-27T01:09:31.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 96199 12.90 MB/sec execute 53 sec latency 44.811 ms 2024-04-27T01:09:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:32 smithi104 ceph-mon[21811]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.8 MiB/s wr, 677 op/s 2024-04-27T01:09:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:32 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:32 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:32.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:32 smithi135 ceph-mon[24080]: pgmap v31: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.8 MiB/s wr, 677 op/s 2024-04-27T01:09:32.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:32 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:32.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:32 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:32.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 96471 12.72 MB/sec execute 54 sec latency 61.905 ms 2024-04-27T01:09:33.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:33 smithi104 ceph-mon[21811]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-04-27T01:09:33.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:33 smithi104 ceph-mon[21811]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-04-27T01:09:33.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:33 smithi135 ceph-mon[24080]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-04-27T01:09:33.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:33 smithi135 ceph-mon[24080]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-04-27T01:09:33.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 96846 12.59 MB/sec execute 55 sec latency 42.624 ms 2024-04-27T01:09:34.129 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:34 smithi104 ceph-mon[21811]: pgmap v32: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 758 op/s 2024-04-27T01:09:34.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:34 smithi135 ceph-mon[24080]: pgmap v32: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 758 op/s 2024-04-27T01:09:34.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 97177 12.47 MB/sec execute 56 sec latency 84.268 ms 2024-04-27T01:09:35.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 97638 12.47 MB/sec execute 57 sec latency 54.030 ms 2024-04-27T01:09:36.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:36 smithi104 ceph-mon[21811]: pgmap v33: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.1 MiB/s wr, 549 op/s 2024-04-27T01:09:36.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:36.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:36 smithi135 ceph-mon[24080]: pgmap v33: 97 pgs: 97 active+clean; 326 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.1 MiB/s wr, 549 op/s 2024-04-27T01:09:36.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:36.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 98204 12.46 MB/sec execute 58 sec latency 38.250 ms 2024-04-27T01:09:37.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 98691 12.56 MB/sec execute 59 sec latency 30.182 ms 2024-04-27T01:09:38.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:38 smithi104 ceph-mon[21811]: pgmap v34: 97 pgs: 97 active+clean; 322 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 796 op/s 2024-04-27T01:09:38.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:38 smithi135 ceph-mon[24080]: pgmap v34: 97 pgs: 97 active+clean; 322 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 796 op/s 2024-04-27T01:09:38.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 99111 12.62 MB/sec execute 60 sec latency 31.005 ms 2024-04-27T01:09:39.475 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:39 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:39.475 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:39 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:39.475 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:39 smithi104 ceph-mon[21811]: Reconfiguring prometheus.smithi104 (dependencies changed)... 2024-04-27T01:09:39.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 99467 12.65 MB/sec execute 61 sec latency 61.646 ms 2024-04-27T01:09:39.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:39 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:39.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:39 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:09:39.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:39 smithi135 ceph-mon[24080]: Reconfiguring prometheus.smithi104 (dependencies changed)... 2024-04-27T01:09:40.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:40 smithi104 ceph-mon[21811]: Reconfiguring daemon prometheus.smithi104 on smithi104 2024-04-27T01:09:40.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:40 smithi104 ceph-mon[21811]: pgmap v35: 97 pgs: 97 active+clean; 322 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.1 MiB/s wr, 533 op/s 2024-04-27T01:09:40.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 99754 12.57 MB/sec execute 62 sec latency 82.808 ms 2024-04-27T01:09:40.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:40 smithi135 ceph-mon[24080]: Reconfiguring daemon prometheus.smithi104 on smithi104 2024-04-27T01:09:40.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:40 smithi135 ceph-mon[24080]: pgmap v35: 97 pgs: 97 active+clean; 322 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.1 MiB/s wr, 533 op/s 2024-04-27T01:09:41.617 INFO:teuthology.orchestra.run.smithi104.stdout: 5 100049 12.43 MB/sec execute 63 sec latency 48.380 ms 2024-04-27T01:09:42.497 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:42 smithi104 ceph-mon[21811]: pgmap v36: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 644 op/s 2024-04-27T01:09:42.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 100420 12.33 MB/sec execute 64 sec latency 54.066 ms 2024-04-27T01:09:42.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:42 smithi135 ceph-mon[24080]: pgmap v36: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 644 op/s 2024-04-27T01:09:43.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 100839 12.32 MB/sec execute 65 sec latency 31.082 ms 2024-04-27T01:09:44.526 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:44 smithi104 ceph-mon[21811]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 633 op/s 2024-04-27T01:09:44.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 101326 12.28 MB/sec execute 66 sec latency 50.771 ms 2024-04-27T01:09:44.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:44 smithi135 ceph-mon[24080]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 633 op/s 2024-04-27T01:09:45.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 101852 12.25 MB/sec execute 67 sec latency 48.756 ms 2024-04-27T01:09:46.042 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:45 smithi104 ceph-mon[21811]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 482 op/s 2024-04-27T01:09:46.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:45 smithi135 ceph-mon[24080]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 482 op/s 2024-04-27T01:09:46.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 102266 12.32 MB/sec execute 68 sec latency 42.318 ms 2024-04-27T01:09:47.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 102805 12.49 MB/sec execute 69 sec latency 36.977 ms 2024-04-27T01:09:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:48 smithi104 ceph-mon[21811]: pgmap v39: 97 pgs: 97 active+clean; 319 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 743 op/s 2024-04-27T01:09:48.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:48 smithi135 ceph-mon[24080]: pgmap v39: 97 pgs: 97 active+clean; 319 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 743 op/s 2024-04-27T01:09:48.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 103229 12.49 MB/sec execute 70 sec latency 32.333 ms 2024-04-27T01:09:49.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 103569 12.37 MB/sec execute 71 sec latency 60.139 ms 2024-04-27T01:09:50.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:50 smithi104 ceph-mon[21811]: pgmap v40: 97 pgs: 97 active+clean; 319 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-04-27T01:09:50.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:50 smithi135 ceph-mon[24080]: pgmap v40: 97 pgs: 97 active+clean; 319 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.2 MiB/s wr, 497 op/s 2024-04-27T01:09:50.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 103957 12.28 MB/sec execute 72 sec latency 43.182 ms 2024-04-27T01:09:51.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:51 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:51.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:51 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:09:51.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 104358 12.24 MB/sec execute 73 sec latency 35.891 ms 2024-04-27T01:09:52.114 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:52 smithi104 ceph-mon[21811]: pgmap v41: 97 pgs: 97 active+clean; 317 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.3 MiB/s wr, 550 op/s 2024-04-27T01:09:52.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:52 smithi135 ceph-mon[24080]: pgmap v41: 97 pgs: 97 active+clean; 317 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.3 MiB/s wr, 550 op/s 2024-04-27T01:09:52.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 104786 12.20 MB/sec execute 74 sec latency 46.998 ms 2024-04-27T01:09:53.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 105229 12.19 MB/sec execute 75 sec latency 197.206 ms 2024-04-27T01:09:54.142 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:09:54.147 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:54 smithi104 ceph-mon[21811]: pgmap v42: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.8 MiB/s wr, 645 op/s 2024-04-27T01:09:54.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:54 smithi135 ceph-mon[24080]: pgmap v42: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.8 MiB/s wr, 645 op/s 2024-04-27T01:09:54.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 105686 12.21 MB/sec execute 76 sec latency 42.869 ms 2024-04-27T01:09:55.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:55 smithi104 ceph-mon[21811]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:55.311 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:09:55.311 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 starting - - - - 2024-04-27T01:09:55.311 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (9m) 65s ago 9m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:09:55.311 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (9m) 65s ago 9m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 starting - - - - 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (3m) 65s ago 3m 38.9M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (3m) 65s ago 3m 15.1M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:9283 running (10m) 65s ago 10m 349M - 17.2.0 e1d6a67b021e 2392eca52c19 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (90s) 65s ago 9m 488M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 cd90a4a984a6 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (10m) 65s ago 11m 45.8M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (8m) 65s ago 8m 40.0M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (3m) 65s ago 3m 265M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (3m) 65s ago 3m 73.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 starting - - - - 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 65s ago 8m 21.2M - 1.3.1 1dbe0e931976 10a43cc6931b 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (8m) 65s ago 8m 293M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:09:55.312 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (7m) 65s ago 7m 234M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (7m) 65s ago 7m 208M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (6m) 65s ago 6m 255M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (6m) 65s ago 6m 266M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (6m) 65s ago 6m 299M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (5m) 65s ago 5m 268M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (5m) 65s ago 5m 353M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:09:55.313 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (8m) 65s ago 8m 65.8M - 2.33.4 514e6a882f6e e619d633de42 2024-04-27T01:09:55.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:55 smithi135 ceph-mon[24080]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:55.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 106332 12.39 MB/sec execute 77 sec latency 25.133 ms 2024-04-27T01:09:55.911 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:09:55.912 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:09:55.913 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-27T01:09:55.913 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:09:55.913 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:09:55.913 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:09:56.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:56 smithi104 ceph-mon[21811]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:56.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:56 smithi104 ceph-mon[21811]: pgmap v43: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T01:09:56.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:56 smithi104 ceph-mon[21811]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:56.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:56 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2053737100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:09:56.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:56 smithi135 ceph-mon[24080]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:56.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:56 smithi135 ceph-mon[24080]: pgmap v43: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 520 op/s 2024-04-27T01:09:56.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:56 smithi135 ceph-mon[24080]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:56.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:56 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2053737100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "1/23 daemons upgraded", 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:09:56.394 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:09:56.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 106704 12.39 MB/sec execute 78 sec latency 89.779 ms 2024-04-27T01:09:56.907 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:09:57.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:57 smithi104 ceph-mon[21811]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:57.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:57 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1774741808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:09:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:57 smithi135 ceph-mon[24080]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:57 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1774741808' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:09:57.618 INFO:teuthology.orchestra.run.smithi104.stdout: 5 106989 12.32 MB/sec execute 79 sec latency 72.934 ms 2024-04-27T01:09:58.237 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:09:58 smithi104 ceph-mon[21811]: pgmap v44: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 791 op/s 2024-04-27T01:09:58.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:09:58 smithi135 ceph-mon[24080]: pgmap v44: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 791 op/s 2024-04-27T01:09:58.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 107561 12.27 MB/sec execute 80 sec latency 31.410 ms 2024-04-27T01:09:59.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 108215 12.31 MB/sec execute 81 sec latency 37.673 ms 2024-04-27T01:10:00.260 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:00 smithi104 ceph-mon[21811]: pgmap v45: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 529 op/s 2024-04-27T01:10:00.260 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:00 smithi104 ceph-mon[21811]: overall HEALTH_OK 2024-04-27T01:10:00.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:00 smithi135 ceph-mon[24080]: pgmap v45: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 529 op/s 2024-04-27T01:10:00.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:00 smithi135 ceph-mon[24080]: overall HEALTH_OK 2024-04-27T01:10:00.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 108848 12.33 MB/sec execute 82 sec latency 35.753 ms 2024-04-27T01:10:01.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 109244 12.33 MB/sec execute 83 sec latency 38.307 ms 2024-04-27T01:10:02.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:02 smithi104 ceph-mon[21811]: pgmap v46: 97 pgs: 97 active+clean; 313 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.6 MiB/s wr, 573 op/s 2024-04-27T01:10:02.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:02 smithi135 ceph-mon[24080]: pgmap v46: 97 pgs: 97 active+clean; 313 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.6 MiB/s wr, 573 op/s 2024-04-27T01:10:02.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 109640 12.37 MB/sec execute 84 sec latency 35.624 ms 2024-04-27T01:10:03.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:03 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:03.586 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:03 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:03.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 110015 12.43 MB/sec execute 85 sec latency 48.846 ms 2024-04-27T01:10:03.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:03 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:03.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:03 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:04.609 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:04 smithi104 ceph-mon[21811]: Reconfiguring node-exporter.smithi135 (dependencies changed)... 2024-04-27T01:10:04.609 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:04 smithi104 ceph-mon[21811]: Reconfiguring daemon node-exporter.smithi135 on smithi135 2024-04-27T01:10:04.609 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:04 smithi104 ceph-mon[21811]: pgmap v47: 97 pgs: 97 active+clean; 319 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 783 op/s 2024-04-27T01:10:04.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 110382 12.39 MB/sec execute 86 sec latency 42.343 ms 2024-04-27T01:10:04.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:04 smithi135 ceph-mon[24080]: Reconfiguring node-exporter.smithi135 (dependencies changed)... 2024-04-27T01:10:04.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:04 smithi135 ceph-mon[24080]: Reconfiguring daemon node-exporter.smithi135 on smithi135 2024-04-27T01:10:04.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:04 smithi135 ceph-mon[24080]: pgmap v47: 97 pgs: 97 active+clean; 319 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.5 MiB/s wr, 783 op/s 2024-04-27T01:10:05.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 110647 12.29 MB/sec execute 87 sec latency 38.874 ms 2024-04-27T01:10:06.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: pgmap v48: 97 pgs: 97 active+clean; 319 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 578 op/s 2024-04-27T01:10:06.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:06.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:10:06.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:06.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:05 smithi104 ceph-mon[21811]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:06.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: pgmap v48: 97 pgs: 97 active+clean; 319 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 578 op/s 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104.front.sepia.ceph.com:9093"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:10:06.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:06.367 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:05 smithi135 ceph-mon[24080]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:06.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 110971 12.22 MB/sec execute 88 sec latency 45.882 ms 2024-04-27T01:10:07.624 INFO:teuthology.orchestra.run.smithi104.stdout: 5 111434 12.18 MB/sec execute 89 sec latency 33.346 ms 2024-04-27T01:10:08.220 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:08 smithi104 ceph-mon[21811]: mgrmap e27: smithi135.nwoumo(active, since 92s), standbys: smithi104.dpemzn 2024-04-27T01:10:08.220 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:08 smithi104 ceph-mon[21811]: pgmap v49: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 794 op/s 2024-04-27T01:10:08.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:08 smithi135 ceph-mon[24080]: mgrmap e27: smithi135.nwoumo(active, since 92s), standbys: smithi104.dpemzn 2024-04-27T01:10:08.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:08 smithi135 ceph-mon[24080]: pgmap v49: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 794 op/s 2024-04-27T01:10:08.627 INFO:teuthology.orchestra.run.smithi104.stdout: 5 111844 12.15 MB/sec execute 90 sec latency 64.649 ms 2024-04-27T01:10:09.627 INFO:teuthology.orchestra.run.smithi104.stdout: 5 112336 12.15 MB/sec execute 91 sec latency 35.287 ms 2024-04-27T01:10:09.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: Upgrade: Updating mgr.smithi104.dpemzn 2024-04-27T01:10:09.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:09.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:10:09.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:10:09.742 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:10:09.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:09.743 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:09 smithi104 ceph-mon[21811]: Deploying daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:10:09.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: Upgrade: Updating mgr.smithi104.dpemzn 2024-04-27T01:10:09.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:09.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:10:09.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:10:09.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:10:09.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:09.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:09 smithi135 ceph-mon[24080]: Deploying daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:10:10.627 INFO:teuthology.orchestra.run.smithi104.stdout: 5 112820 12.20 MB/sec execute 92 sec latency 30.884 ms 2024-04-27T01:10:10.765 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:10 smithi104 ceph-mon[21811]: pgmap v50: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 523 op/s 2024-04-27T01:10:10.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:10 smithi135 ceph-mon[24080]: pgmap v50: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.4 MiB/s wr, 523 op/s 2024-04-27T01:10:11.627 INFO:teuthology.orchestra.run.smithi104.stdout: 5 113235 12.21 MB/sec execute 93 sec latency 31.369 ms 2024-04-27T01:10:12.627 INFO:teuthology.orchestra.run.smithi104.stdout: 5 113664 12.27 MB/sec execute 94 sec latency 48.766 ms 2024-04-27T01:10:12.794 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:12 smithi104 ceph-mon[21811]: pgmap v51: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.2 MiB/s wr, 576 op/s 2024-04-27T01:10:12.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:12 smithi135 ceph-mon[24080]: pgmap v51: 97 pgs: 97 active+clean; 326 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.2 MiB/s wr, 576 op/s 2024-04-27T01:10:13.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 113978 12.24 MB/sec execute 95 sec latency 48.146 ms 2024-04-27T01:10:14.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 114377 12.18 MB/sec execute 96 sec latency 56.328 ms 2024-04-27T01:10:14.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:14 smithi104 ceph-mon[21811]: pgmap v52: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 748 op/s 2024-04-27T01:10:14.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:14 smithi135 ceph-mon[24080]: pgmap v52: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 748 op/s 2024-04-27T01:10:15.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 114890 12.12 MB/sec execute 97 sec latency 53.395 ms 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: pgmap v53: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.579 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:16 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 115413 12.13 MB/sec execute 98 sec latency 28.962 ms 2024-04-27T01:10:16.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: pgmap v53: 97 pgs: 97 active+clean; 323 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.0 MiB/s wr, 485 op/s 2024-04-27T01:10:16.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:16.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:16.866 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:16 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:17.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 115964 12.14 MB/sec execute 99 sec latency 38.305 ms 2024-04-27T01:10:18.619 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:18 smithi104 ceph-mon[21811]: pgmap v54: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 748 op/s 2024-04-27T01:10:18.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 116488 12.23 MB/sec execute 100 sec latency 34.257 ms 2024-04-27T01:10:18.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:18 smithi135 ceph-mon[24080]: pgmap v54: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 748 op/s 2024-04-27T01:10:19.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 116926 12.26 MB/sec execute 101 sec latency 29.803 ms 2024-04-27T01:10:20.406 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:20 smithi104 ceph-mon[21811]: pgmap v55: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 532 op/s 2024-04-27T01:10:20.406 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:20 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:20.406 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:20 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:20.406 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:20 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:20.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 117360 12.31 MB/sec execute 102 sec latency 39.054 ms 2024-04-27T01:10:20.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:20 smithi135 ceph-mon[24080]: pgmap v55: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 532 op/s 2024-04-27T01:10:20.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:20 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:20.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:20 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:20.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:20 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:21.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 117672 12.23 MB/sec execute 103 sec latency 35.028 ms 2024-04-27T01:10:22.418 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:22 smithi104 ceph-mon[21811]: pgmap v56: 97 pgs: 97 active+clean; 326 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.5 MiB/s wr, 631 op/s 2024-04-27T01:10:22.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 118328 12.22 MB/sec execute 104 sec latency 28.878 ms 2024-04-27T01:10:22.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:22 smithi135 ceph-mon[24080]: pgmap v56: 97 pgs: 97 active+clean; 326 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.5 MiB/s wr, 631 op/s 2024-04-27T01:10:23.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 119096 12.27 MB/sec execute 105 sec latency 23.169 ms 2024-04-27T01:10:24.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:24 smithi104 ceph-mon[21811]: pgmap v57: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 750 op/s 2024-04-27T01:10:24.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 119753 12.30 MB/sec execute 106 sec latency 40.480 ms 2024-04-27T01:10:24.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:24 smithi135 ceph-mon[24080]: pgmap v57: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 750 op/s 2024-04-27T01:10:25.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 120265 12.41 MB/sec execute 107 sec latency 37.796 ms 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: pgmap v58: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: Standby manager daemon smithi104.dpemzn restarted 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: Standby manager daemon smithi104.dpemzn started 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/crt"}]: dispatch 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/key"}]: dispatch 2024-04-27T01:10:26.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:26 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:26.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: pgmap v58: 97 pgs: 97 active+clean; 327 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-04-27T01:10:26.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: Standby manager daemon smithi104.dpemzn restarted 2024-04-27T01:10:26.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: Standby manager daemon smithi104.dpemzn started 2024-04-27T01:10:26.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/crt"}]: dispatch 2024-04-27T01:10:26.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:26.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.dpemzn/key"}]: dispatch 2024-04-27T01:10:26.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:26 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:26.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 120735 12.45 MB/sec execute 108 sec latency 43.049 ms 2024-04-27T01:10:27.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:27 smithi104 ceph-mon[21811]: mgrmap e28: smithi135.nwoumo(active, since 111s), standbys: smithi104.dpemzn 2024-04-27T01:10:27.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:27 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:27.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:27 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:27.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:27 smithi135 ceph-mon[24080]: mgrmap e28: smithi135.nwoumo(active, since 111s), standbys: smithi104.dpemzn 2024-04-27T01:10:27.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:27 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:27.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:27 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:27.514 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:10:27.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 121048 12.42 MB/sec execute 109 sec latency 37.766 ms 2024-04-27T01:10:28.508 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (77s) 2s ago 10m 24.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (10m) 2s ago 10m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (9m) 8s ago 9m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 unknown 2s ago 9m - - 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (4m) 2s ago 4m 39.3M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (4m) 8s ago 4m 15.7M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (14s) 2s ago 11m 141M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (2m) 8s ago 9m 507M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 cd90a4a984a6 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (11m) 2s ago 11m 49.0M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (9m) 8s ago 9m 43.3M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (4m) 2s ago 4m 267M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (3m) 8s ago 3m 79.0M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:10:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (50s) 2s ago 9m 10.2M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (23s) 8s ago 9m 9210k - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (8m) 2s ago 8m 451M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (8m) 2s ago 8m 361M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (7m) 2s ago 7m 302M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (7m) 2s ago 7m 403M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (7m) 8s ago 7m 391M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (6m) 8s ago 6m 438M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (6m) 8s ago 6m 381M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (5m) 8s ago 5m 495M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:10:28.510 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (26s) 2s ago 9m 51.1M - 2.33.4 514e6a882f6e 4f4a8c23db05 2024-04-27T01:10:28.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 121411 12.34 MB/sec execute 110 sec latency 28.209 ms 2024-04-27T01:10:28.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:28 smithi104 ceph-mon[21811]: pgmap v59: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 880 op/s 2024-04-27T01:10:28.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:28 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:28.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:28 smithi104 ceph-mon[21811]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:28.740 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:28 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:28.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:28 smithi135 ceph-mon[24080]: pgmap v59: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 880 op/s 2024-04-27T01:10:28.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:28 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:28.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:28 smithi135 ceph-mon[24080]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:28.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:28 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:10:29.105 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:10:29.106 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:10:29.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:29 smithi104 ceph-mon[21811]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:29.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:29 smithi104 ceph-mon[21811]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:29.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:29 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/683022317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:10:29.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 121923 12.31 MB/sec execute 111 sec latency 30.249 ms 2024-04-27T01:10:29.696 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading mgr daemons", 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:10:29.697 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:10:29.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:29 smithi135 ceph-mon[24080]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:29.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:29 smithi135 ceph-mon[24080]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:29.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:29 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/683022317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:10:30.309 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:10:30.525 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:30 smithi104 ceph-mon[21811]: pgmap v60: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-04-27T01:10:30.525 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:30 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/1896299293' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:10:30.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 122374 12.31 MB/sec execute 112 sec latency 56.058 ms 2024-04-27T01:10:30.766 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:30 smithi135 ceph-mon[24080]: pgmap v60: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 617 op/s 2024-04-27T01:10:30.766 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:30 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/1896299293' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:10:31.551 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:31 smithi104 ceph-mon[21811]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:31.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 122933 12.30 MB/sec execute 113 sec latency 35.167 ms 2024-04-27T01:10:31.777 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:31 smithi135 ceph-mon[24080]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:32.629 INFO:teuthology.orchestra.run.smithi104.stdout: 5 123285 12.27 MB/sec execute 114 sec latency 48.873 ms 2024-04-27T01:10:32.655 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:32 smithi104 ceph-mon[21811]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.1 MiB/s wr, 805 op/s 2024-04-27T01:10:32.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:32 smithi135 ceph-mon[24080]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.1 MiB/s wr, 805 op/s 2024-04-27T01:10:33.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 123787 12.38 MB/sec execute 115 sec latency 41.960 ms 2024-04-27T01:10:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 124398 12.44 MB/sec execute 116 sec latency 32.848 ms 2024-04-27T01:10:34.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:34 smithi104 ceph-mon[21811]: pgmap v62: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 782 op/s 2024-04-27T01:10:34.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:34 smithi135 ceph-mon[24080]: pgmap v62: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.7 MiB/s wr, 782 op/s 2024-04-27T01:10:35.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 124695 12.40 MB/sec execute 117 sec latency 32.837 ms 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: pgmap v63: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:36.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: pgmap v64: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 714 op/s 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: Failing over to other MGR 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:36.207 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:36 smithi104 ceph-mon[21811]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: pgmap v63: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.9 MiB/s wr, 609 op/s 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:36.266 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: pgmap v64: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 714 op/s 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: Upgrade: Need to upgrade myself (mgr.smithi135.nwoumo) 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: Failing over to other MGR 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 172.21.15.135:0/1430422436' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:36.267 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:36 smithi135 ceph-mon[24080]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:10:36.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 125002 12.34 MB/sec execute 118 sec latency 48.462 ms 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd='[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]': finished 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: mgrmap e29: smithi104.dpemzn(active, starting, since 0.858499s) 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:10:36.999 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:10:37.000 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:37.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:37.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:37.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:37.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:37 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.24415 ' entity='mgr.smithi135.nwoumo' cmd='[{"prefix": "mgr fail", "who": "smithi135.nwoumo"}]': finished 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: mgrmap e29: smithi104.dpemzn(active, starting, since 0.858499s) 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:10:37.340 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:10:37.341 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:37.342 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:37.343 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:37 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:37.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 125232 12.25 MB/sec execute 119 sec latency 115.124 ms 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: [27/Apr/2024:01:10:37] ENGINE Bus STARTING 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: [27/Apr/2024:01:10:37] ENGINE Serving on http://172.21.15.104:8765 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: mgrmap e30: smithi104.dpemzn(active, since 1.89054s) 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: [27/Apr/2024:01:10:37] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: [27/Apr/2024:01:10:37] ENGINE Bus STARTED 2024-04-27T01:10:38.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:38 smithi104 ceph-mon[21811]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: [27/Apr/2024:01:10:37] ENGINE Bus STARTING 2024-04-27T01:10:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: [27/Apr/2024:01:10:37] ENGINE Serving on http://172.21.15.104:8765 2024-04-27T01:10:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: mgrmap e30: smithi104.dpemzn(active, since 1.89054s) 2024-04-27T01:10:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: [27/Apr/2024:01:10:37] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:10:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: [27/Apr/2024:01:10:37] ENGINE Bus STARTED 2024-04-27T01:10:38.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:38 smithi135 ceph-mon[24080]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:38.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 125507 12.19 MB/sec execute 120 sec latency 52.935 ms 2024-04-27T01:10:39.362 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:39 smithi104 ceph-mon[21811]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:39.362 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:39 smithi104 ceph-mon[21811]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:10:39.362 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:39 smithi104 ceph-mon[21811]: Cluster is now healthy 2024-04-27T01:10:39.362 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:39 smithi104 ceph-mon[21811]: mgrmap e31: smithi104.dpemzn(active, since 2s) 2024-04-27T01:10:39.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:39 smithi135 ceph-mon[24080]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:39.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:39 smithi135 ceph-mon[24080]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:10:39.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:39 smithi135 ceph-mon[24080]: Cluster is now healthy 2024-04-27T01:10:39.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:39 smithi135 ceph-mon[24080]: mgrmap e31: smithi104.dpemzn(active, since 2s) 2024-04-27T01:10:39.630 INFO:teuthology.orchestra.run.smithi104.stdout: 5 125831 12.18 MB/sec execute 121 sec latency 44.969 ms 2024-04-27T01:10:40.632 INFO:teuthology.orchestra.run.smithi104.stdout: 5 126254 12.15 MB/sec execute 122 sec latency 70.833 ms 2024-04-27T01:10:41.394 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:41 smithi104 ceph-mon[21811]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:41.394 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:41 smithi104 ceph-mon[21811]: mgrmap e32: smithi104.dpemzn(active, since 4s) 2024-04-27T01:10:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:41 smithi135 ceph-mon[24080]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:41 smithi135 ceph-mon[24080]: mgrmap e32: smithi104.dpemzn(active, since 4s) 2024-04-27T01:10:41.632 INFO:teuthology.orchestra.run.smithi104.stdout: 5 126579 12.09 MB/sec execute 123 sec latency 61.454 ms 2024-04-27T01:10:42.632 INFO:teuthology.orchestra.run.smithi104.stdout: 5 126873 12.07 MB/sec execute 124 sec latency 44.625 ms 2024-04-27T01:10:42.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:42 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:42.678 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:42 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:42.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:42 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:42.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:42 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:43.635 INFO:teuthology.orchestra.run.smithi104.stdout: 5 127258 12.11 MB/sec execute 125 sec latency 35.610 ms 2024-04-27T01:10:43.953 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:43 smithi104 ceph-mon[21811]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:43.953 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:43 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:43.953 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:43 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:44.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:43 smithi135 ceph-mon[24080]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:10:44.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:43 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:44.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:43 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:44.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 127627 12.13 MB/sec execute 126 sec latency 42.308 ms 2024-04-27T01:10:44.964 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:44 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:10:44.964 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:10:44.964 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:44.964 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:10:44.964 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:44 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:45.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:44 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:10:45.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:10:45.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:45.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:10:45.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:44 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/477534616' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:45.635 INFO:teuthology.orchestra.run.smithi104.stdout: 5 127993 12.14 MB/sec execute 127 sec latency 62.446 ms 2024-04-27T01:10:45.980 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:45 smithi104 ceph-mon[21811]: pgmap v7: 97 pgs: 97 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 365 op/s 2024-04-27T01:10:45.980 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:45 smithi104 ceph-mon[21811]: mgrmap e33: smithi104.dpemzn(active, since 9s), standbys: smithi135.nwoumo 2024-04-27T01:10:45.980 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:45 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:46.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:45 smithi135 ceph-mon[24080]: pgmap v7: 97 pgs: 97 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 365 op/s 2024-04-27T01:10:46.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:45 smithi135 ceph-mon[24080]: mgrmap e33: smithi104.dpemzn(active, since 9s), standbys: smithi135.nwoumo 2024-04-27T01:10:46.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:45 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:10:46.635 INFO:teuthology.orchestra.run.smithi104.stdout: 5 128344 12.09 MB/sec execute 128 sec latency 36.096 ms 2024-04-27T01:10:47.635 INFO:teuthology.orchestra.run.smithi104.stdout: 5 128737 12.04 MB/sec execute 129 sec latency 36.404 ms 2024-04-27T01:10:47.750 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:47 smithi104 ceph-mon[21811]: pgmap v8: 97 pgs: 97 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-04-27T01:10:47.750 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:47 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:47.750 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:47 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:48.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:47 smithi135 ceph-mon[24080]: pgmap v8: 97 pgs: 97 active+clean; 323 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.3 MiB/s wr, 283 op/s 2024-04-27T01:10:48.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:47 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:48.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:47 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:48.635 INFO:teuthology.orchestra.run.smithi104.stdout: 5 129086 12.01 MB/sec execute 130 sec latency 37.285 ms 2024-04-27T01:10:49.511 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:49 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:49.511 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:49 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:49.511 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:49 smithi104 ceph-mon[21811]: pgmap v9: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-04-27T01:10:49.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:49 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:49.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:49 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:49.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:49 smithi135 ceph-mon[24080]: pgmap v9: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.7 MiB/s wr, 474 op/s 2024-04-27T01:10:49.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 129408 11.99 MB/sec execute 131 sec latency 63.330 ms 2024-04-27T01:10:50.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 129793 11.96 MB/sec execute 132 sec latency 75.267 ms 2024-04-27T01:10:51.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 130189 11.91 MB/sec execute 133 sec latency 64.416 ms 2024-04-27T01:10:52.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:52 smithi104 ceph-mon[21811]: pgmap v10: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.3 MiB/s wr, 432 op/s 2024-04-27T01:10:52.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:52 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:52.071 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:52 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:52.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:52 smithi135 ceph-mon[24080]: pgmap v10: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.3 MiB/s wr, 432 op/s 2024-04-27T01:10:52.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:52 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:52.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:52 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:52.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 130530 11.92 MB/sec execute 134 sec latency 61.243 ms 2024-04-27T01:10:53.639 INFO:teuthology.orchestra.run.smithi104.stdout: 5 130983 11.97 MB/sec execute 135 sec latency 28.979 ms 2024-04-27T01:10:54.158 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:54 smithi104 ceph-mon[21811]: pgmap v11: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.3 MiB/s wr, 432 op/s 2024-04-27T01:10:54.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:54 smithi135 ceph-mon[24080]: pgmap v11: 97 pgs: 97 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.3 MiB/s wr, 432 op/s 2024-04-27T01:10:54.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 131393 12.00 MB/sec execute 136 sec latency 46.113 ms 2024-04-27T01:10:55.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 131729 11.98 MB/sec execute 137 sec latency 39.488 ms 2024-04-27T01:10:56.223 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:56 smithi104 ceph-mon[21811]: pgmap v12: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 655 op/s 2024-04-27T01:10:56.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:56 smithi135 ceph-mon[24080]: pgmap v12: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 655 op/s 2024-04-27T01:10:56.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 132100 11.93 MB/sec execute 138 sec latency 36.485 ms 2024-04-27T01:10:57.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 132351 11.86 MB/sec execute 139 sec latency 64.652 ms 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 444 op/s 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:57.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:57 smithi104 ceph-mon[21811]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: pgmap v13: 97 pgs: 97 active+clean; 325 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 444 op/s 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:58.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:57 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:58.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 132594 11.81 MB/sec execute 140 sec latency 45.277 ms 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.839 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 695 op/s 2024-04-27T01:10:58.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:58.840 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:58 smithi104 ceph-mon[21811]: Reconfiguring prometheus.smithi104 (dependencies changed)... 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: pgmap v14: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 695 op/s 2024-04-27T01:10:59.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:10:59.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:58 smithi135 ceph-mon[24080]: Reconfiguring prometheus.smithi104 (dependencies changed)... 2024-04-27T01:10:59.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 132855 11.79 MB/sec execute 141 sec latency 51.843 ms 2024-04-27T01:10:59.857 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:59 smithi104 ceph-mon[21811]: Reconfiguring daemon prometheus.smithi104 on smithi104 2024-04-27T01:10:59.858 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:10:59 smithi104 ceph-mon[21811]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:11:00.016 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:59 smithi135 ceph-mon[24080]: Reconfiguring daemon prometheus.smithi104 on smithi104 2024-04-27T01:11:00.016 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:10:59 smithi135 ceph-mon[24080]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:11:00.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 133143 11.76 MB/sec execute 142 sec latency 86.956 ms 2024-04-27T01:11:00.733 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:11:01.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:01 smithi104 ceph-mon[21811]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-04-27T01:11:01.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:01 smithi135 ceph-mon[24080]: pgmap v15: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-04-27T01:11:01.605 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:11:01.605 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (110s) 14s ago 10m 24.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:11:01.605 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (10m) 14s ago 10m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:11:01.605 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (10m) 20s ago 10m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 unknown 14s ago 10m - - 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (4m) 14s ago 4m 39.4M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (4m) 20s ago 4m 15.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (47s) 14s ago 11m 474M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283 running (2m) 20s ago 10m 70.6M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 cd90a4a984a6 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (12m) 14s ago 12m 53.4M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (10m) 20s ago 10m 47.4M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (4m) 14s ago 4m 276M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (4m) 20s ago 4m 79.3M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (83s) 14s ago 10m 18.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (56s) 20s ago 9m 8400k - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (9m) 14s ago 9m 461M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:11:01.606 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (8m) 14s ago 8m 396M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:11:01.607 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (8m) 14s ago 8m 322M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:11:01.607 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (7m) 14s ago 7m 412M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:11:01.607 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (7m) 20s ago 7m 433M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:11:01.609 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (7m) 20s ago 7m 469M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:11:01.609 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (6m) 20s ago 6m 413M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:11:01.609 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (6m) 20s ago 6m 532M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:11:01.609 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (59s) 14s ago 9m 53.6M - 2.33.4 514e6a882f6e 4f4a8c23db05 2024-04-27T01:11:01.638 INFO:teuthology.orchestra.run.smithi104.stdout: 5 133577 11.73 MB/sec execute 143 sec latency 102.303 ms 2024-04-27T01:11:02.197 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:02 smithi104 ceph-mon[21811]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:11:02.267 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:11:02.268 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:11:02.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:02 smithi135 ceph-mon[24080]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:02.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 134079 11.75 MB/sec execute 144 sec latency 24.217 ms 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:11:02.744 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:11:02.745 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:11:03.141 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:03 smithi104 ceph-mon[21811]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:03.142 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:03 smithi104 ceph-mon[21811]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:03.142 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:03 smithi104 ceph-mon[21811]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-04-27T01:11:03.142 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:03 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/4135470672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:03.317 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:11:03.317 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:11:03.317 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in unknown state 2024-04-27T01:11:03.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:03 smithi135 ceph-mon[24080]: from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:03.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:03 smithi135 ceph-mon[24080]: from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:03.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:03 smithi135 ceph-mon[24080]: pgmap v16: 97 pgs: 97 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 464 op/s 2024-04-27T01:11:03.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:03 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/4135470672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:03.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 134534 11.81 MB/sec execute 145 sec latency 40.365 ms 2024-04-27T01:11:04.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:04 smithi104 ceph-mon[21811]: from='client.24575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:04.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:04 smithi104 ceph-mon[21811]: from='client.? 172.21.15.104:0/2306214218' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:11:04.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:04 smithi135 ceph-mon[24080]: from='client.24575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:04.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:04 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2306214218' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:11:04.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 134929 11.81 MB/sec execute 146 sec latency 43.447 ms 2024-04-27T01:11:05.190 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:05 smithi104 ceph-mon[21811]: pgmap v17: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 676 op/s 2024-04-27T01:11:05.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:05 smithi135 ceph-mon[24080]: pgmap v17: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 676 op/s 2024-04-27T01:11:05.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 135226 11.81 MB/sec execute 147 sec latency 43.402 ms 2024-04-27T01:11:06.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 135465 11.75 MB/sec execute 148 sec latency 64.534 ms 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: pgmap v18: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.022 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:06 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: pgmap v18: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:07.231 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:06 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:07.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 135828 11.70 MB/sec execute 149 sec latency 59.501 ms 2024-04-27T01:11:08.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 136203 11.69 MB/sec execute 150 sec latency 25.068 ms 2024-04-27T01:11:09.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:08 smithi135 ceph-mon[24080]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 670 op/s 2024-04-27T01:11:09.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:08 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:11:09.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:08 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:11:09.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:08 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:09.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:08 smithi104 ceph-mon[21811]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.1 MiB/s wr, 670 op/s 2024-04-27T01:11:09.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:08 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:11:09.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:08 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:11:09.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:08 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:09.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 136565 11.67 MB/sec execute 151 sec latency 51.001 ms 2024-04-27T01:11:10.024 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:09 smithi104 ceph-mon[21811]: Upgrade: Updating mgr.smithi135.nwoumo 2024-04-27T01:11:10.025 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:09 smithi104 ceph-mon[21811]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:11:10.178 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:09 smithi135 ceph-mon[24080]: Upgrade: Updating mgr.smithi135.nwoumo 2024-04-27T01:11:10.178 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:09 smithi135 ceph-mon[24080]: Deploying daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:11:10.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 137278 11.69 MB/sec execute 152 sec latency 30.817 ms 2024-04-27T01:11:11.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:11 smithi104 ceph-mon[21811]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-27T01:11:11.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:11 smithi135 ceph-mon[24080]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-27T01:11:11.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 137895 11.77 MB/sec execute 153 sec latency 30.441 ms 2024-04-27T01:11:12.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 138547 11.85 MB/sec execute 154 sec latency 29.334 ms 2024-04-27T01:11:13.050 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:13 smithi104 ceph-mon[21811]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-27T01:11:13.051 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:13 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:13.051 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:13 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:13.051 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:13 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:13.329 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:13 smithi135 ceph-mon[24080]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-04-27T01:11:13.329 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:13 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:13.329 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:13 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:13.330 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:13 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:13.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 138904 11.83 MB/sec execute 155 sec latency 38.562 ms 2024-04-27T01:11:14.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:14 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:14.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:14 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:14.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:14 smithi104 ceph-mon[21811]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 731 op/s 2024-04-27T01:11:14.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:14 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:14.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:14 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:14.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:14 smithi135 ceph-mon[24080]: pgmap v22: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 731 op/s 2024-04-27T01:11:14.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 139391 11.79 MB/sec execute 156 sec latency 47.654 ms 2024-04-27T01:11:15.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 139933 11.81 MB/sec execute 157 sec latency 41.967 ms 2024-04-27T01:11:16.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 140489 11.81 MB/sec execute 158 sec latency 31.293 ms 2024-04-27T01:11:17.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:17 smithi104 ceph-mon[21811]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.7 MiB/s wr, 528 op/s 2024-04-27T01:11:17.181 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:17 smithi135 ceph-mon[24080]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.7 MiB/s wr, 528 op/s 2024-04-27T01:11:17.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 140963 11.79 MB/sec execute 159 sec latency 45.132 ms 2024-04-27T01:11:18.616 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:18 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:18.617 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:18 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:18.617 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:18 smithi104 ceph-mon[21811]: pgmap v24: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 833 op/s 2024-04-27T01:11:18.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 141358 11.83 MB/sec execute 160 sec latency 40.013 ms 2024-04-27T01:11:18.799 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:18 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:18.799 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:18 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:18.799 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:18 smithi135 ceph-mon[24080]: pgmap v24: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 833 op/s 2024-04-27T01:11:19.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 141956 11.89 MB/sec execute 161 sec latency 29.722 ms 2024-04-27T01:11:20.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:19 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:20.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:19 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:20.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:19 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:20.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:19 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:20.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 142370 11.90 MB/sec execute 162 sec latency 29.781 ms 2024-04-27T01:11:21.155 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:21 smithi104 ceph-mon[21811]: pgmap v25: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 616 op/s 2024-04-27T01:11:21.257 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:21 smithi135 ceph-mon[24080]: pgmap v25: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 616 op/s 2024-04-27T01:11:21.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 142805 11.87 MB/sec execute 163 sec latency 47.289 ms 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:22.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:22 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:11:22.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.? 172.21.15.135:0/2698859697' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:11:22.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:22.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:22.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:22 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:22.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 143266 11.85 MB/sec execute 164 sec latency 31.941 ms 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 616 op/s 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: mgrmap e34: smithi104.dpemzn(active, since 46s), standbys: smithi135.nwoumo 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:23 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: pgmap v26: 97 pgs: 97 active+clean; 325 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.9 MiB/s wr, 616 op/s 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: mgrmap e34: smithi104.dpemzn(active, since 46s), standbys: smithi135.nwoumo 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:23 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:23.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 143713 11.84 MB/sec execute 165 sec latency 38.295 ms 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 961 op/s 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:24.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.dpemzn"}]: dispatch 2024-04-27T01:11:24.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.dpemzn"}]': finished 2024-04-27T01:11:24.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.nwoumo"}]: dispatch 2024-04-27T01:11:24.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.nwoumo"}]': finished 2024-04-27T01:11:24.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:24 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 961 op/s 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.dpemzn"}]: dispatch 2024-04-27T01:11:24.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.dpemzn"}]': finished 2024-04-27T01:11:24.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.nwoumo"}]: dispatch 2024-04-27T01:11:24.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi135.nwoumo"}]': finished 2024-04-27T01:11:24.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:24 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T01:11:24.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 144361 11.85 MB/sec execute 166 sec latency 36.477 ms 2024-04-27T01:11:25.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:25 smithi104 ceph-mon[21811]: Upgrade: Setting container_image for all mgr 2024-04-27T01:11:25.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:25 smithi135 ceph-mon[24080]: Upgrade: Setting container_image for all mgr 2024-04-27T01:11:25.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 144831 11.89 MB/sec execute 167 sec latency 55.485 ms 2024-04-27T01:11:26.201 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:26 smithi104 ceph-mon[21811]: pgmap v29: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-27T01:11:26.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:26 smithi135 ceph-mon[24080]: pgmap v29: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 763 op/s 2024-04-27T01:11:26.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 145237 11.92 MB/sec execute 168 sec latency 51.456 ms 2024-04-27T01:11:27.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: Upgrade: Updating mon.smithi104 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: Deploying daemon mon.smithi104 on smithi104 2024-04-27T01:11:27.215 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:27 smithi104 ceph-mon[21811]: pgmap v30: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.1 MiB/s wr, 364 op/s 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: Upgrade: Updating mon.smithi104 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: Deploying daemon mon.smithi104 on smithi104 2024-04-27T01:11:27.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:27 smithi135 ceph-mon[24080]: pgmap v30: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.1 MiB/s wr, 364 op/s 2024-04-27T01:11:27.644 INFO:teuthology.orchestra.run.smithi104.stdout: 5 145592 11.91 MB/sec execute 169 sec latency 33.249 ms 2024-04-27T01:11:28.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 145887 11.90 MB/sec execute 170 sec latency 56.384 ms 2024-04-27T01:11:29.052 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:29 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:29.555 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:29 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[21807]: 2024-04-27T01:11:29.465+0000 7fdeab8fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:11:29.555 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:29 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[21807]: 2024-04-27T01:11:29.465+0000 7fdeab8fb700 -1 mon.smithi104@0(leader) e2 *** Got Signal Terminated *** 2024-04-27T01:11:29.555 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:29 smithi104 podman[63766]: 2024-04-27 01:11:29.497776163 +0000 UTC m=+0.213418169 container died 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, release=754, distribution-scope=public, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-27T01:11:29.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 146285 11.87 MB/sec execute 171 sec latency 31.752 ms 2024-04-27T01:11:30.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 podman[63766]: 2024-04-27 01:11:30.120141834 +0000 UTC m=+0.835783840 container cleanup 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, version=8, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-04-27T01:11:30.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 bash[63766]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104 2024-04-27T01:11:30.558 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 podman[63787]: 2024-04-27 01:11:30.506516939 +0000 UTC m=+1.000632077 container remove 8177cde7a2eae88928af457bb408ad11ad241db746aedfc6f005e3eb7b7d991a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, release=754, version=8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public) 2024-04-27T01:11:30.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 146770 11.84 MB/sec execute 172 sec latency 46.078 ms 2024-04-27T01:11:31.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service: Deactivated successfully. 2024-04-27T01:11:31.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 systemd[1]: Stopped Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:31.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:30 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service: Consumed 16.352s CPU time. 2024-04-27T01:11:31.319 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:31 smithi104 systemd[1]: Starting Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:31.570 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:31 smithi104 podman[63908]: 2024-04-27 01:11:31.40186024 +0000 UTC m=+0.028491854 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:11:31.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 147205 11.83 MB/sec execute 173 sec latency 35.531 ms 2024-04-27T01:11:31.822 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:31 smithi104 podman[63908]: 2024-04-27 01:11:31.586072735 +0000 UTC m=+0.212704345 container create 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb) 2024-04-27T01:11:32.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 podman[63908]: 2024-04-27 01:11:32.021259992 +0000 UTC m=+0.647891603 container init 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-27T01:11:32.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 podman[63908]: 2024-04-27 01:11:32.032224941 +0000 UTC m=+0.658856542 container start 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-27T01:11:32.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:11:32.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: pidfile_write: ignore empty --pid-file 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: load: jerasure load: lrc 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Git sha 0 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: DB SUMMARY 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: DB Session ID: 1VKC3I20EIGVVLZRER6W 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-04-27T01:11:32.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi104/store.db dir, Total Num: 1, files: 000030.sst 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi104/store.db: 000028.log size: 2261704 ; 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.env: 0x55e07fc39c60 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.info_log: 0x55e08081d960 2024-04-27T01:11:32.075 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:11:32.076 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.statistics: (nil) 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.use_fsync: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:11:32.077 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.db_log_dir: 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.wal_dir: 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:11:32.078 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.write_buffer_manager: 0x55e0807f01e0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.unordered_write: 0 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:11:32.079 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.row_cache: None 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.wal_filter: None 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.wal_compression: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:11:32.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:11:32.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_open_files: -1 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:11:32.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Compression algorithms supported: 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kZSTD supported: 0 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:11:32.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000015 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.merge_operator: 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_filter: None 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e08081da60) 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:11:32.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_type: 0 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_index_type: 0 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_shortening: 1 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: checksum: 4 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: no_block_cache: 0 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache: 0x55e080821090 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_options: 2024-04-27T01:11:32.085 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: capacity : 536870912 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_shard_bits : 4 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: strict_capacity_limit : 0 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_cache_compressed: (nil) 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: persistent_cache: (nil) 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size: 4096 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_size_deviation: 10 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_restart_interval: 16 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: index_block_restart_interval: 1 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_block_size: 4096 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: partition_filters: 0 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: use_delta_encoding: 1 2024-04-27T01:11:32.086 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: filter_policy: bloomfilter 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: whole_key_filtering: 1 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: verify_compression: 0 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: format_version: 5 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_index_compression: 1 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: block_align: 0 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_auto_readahead_size: 262144 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: prepopulate_block_cache: 0 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:11:32.087 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression: NoCompression 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.num_levels: 7 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:11:32.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:11:32.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:11:32.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:11:32.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:11:32.092 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:11:32.093 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:11:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.ttl: 2592000 2024-04-27T01:11:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:11:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:11:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:11:32.094 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T01:11:32.095 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi104/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11840, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b20ce235-7031-44e0-8824-3deb5867ac6a 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180292072135, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180292081098, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2077985, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11841, "largest_seqno": 12884, "table_properties": {"data_size": 2072747, "index_size": 3071, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 10780, "raw_average_key_size": 23, "raw_value_size": 2062733, "raw_average_value_size": 4523, "num_data_blocks": 142, "num_entries": 456, "num_filter_entries": 456, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714180292, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b20ce235-7031-44e0-8824-3deb5867ac6a", "db_session_id": "1VKC3I20EIGVVLZRER6W", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180292081238, "job": 1, "event": "recovery_finished"} 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi104/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e080912000 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: DB pointer 0x55e0808fc000 2024-04-27T01:11:32.096 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** DB Stats ** 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:11:32.097 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L0 1/0 1.98 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 411.1 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: L6 1/0 6.61 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Sum 2/0 8.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 411.1 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 411.1 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** Compaction Stats [default] ** 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 411.1 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:11:32.099 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Cumulative compaction: 0.00 GB write, 132.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Interval compaction: 0.00 GB write, 132.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Block cache BinnedLRUCache@0x55e080821090#2 capacity: 512.00 MB usage: 4.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:11:32.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: starting mon.smithi104 rank 0 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi104 fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???) e2 preinit fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).mds e7 new map 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).mds e7 print_map 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: e7 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: legacy client fscid: 1 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Filesystem 'foofs' (1) 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: fs_name foofs 2024-04-27T01:11:32.324 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: epoch 5 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: created 2024-04-27T01:06:01.041077+0000 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: modified 2024-04-27T01:06:06.942013+0000 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: tableserver 0 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: root 0 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: session_timeout 60 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: session_autoclose 300 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_file_size 1099511627776 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_xattr_size 65536 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: required_client_features {} 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: last_failure 0 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: last_failure_osd_epoch 0 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: max_mds 1 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: in 0 2024-04-27T01:11:32.325 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: up {0=14492} 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: failed 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: damaged 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: stopped 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: data_pools [3] 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: metadata_pool 2 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: inline_data disabled 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: balancer 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: bal_rank_mask -1 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: standby_count_wanted 1 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: qdb_cluster leader: 0 members: 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.foofs.smithi104.wfnrbv{0:14492} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: Standby daemons: 2024-04-27T01:11:32.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout: [mds.foofs.smithi135.brxmmu{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).osd e75 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 bash[63908]: 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 2024-04-27T01:11:32.327 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 systemd[1]: Started Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:32.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 147793 11.85 MB/sec execute 174 sec latency 31.473 ms 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: pgmap v32: 97 pgs: 97 active+clean; 330 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 730 op/s 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104 calling monitor election 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: monmap epoch 2 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: last_changed 2024-04-27T01:00:59.463740+0000 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: created 2024-04-27T00:58:50.728785+0000 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: min_mon_release 17 (quincy) 2024-04-27T01:11:32.832 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: election_strategy: 1 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mgrmap e34: smithi104.dpemzn(active, since 57s), standbys: smithi135.nwoumo 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: daemon grafana.smithi104 on smithi104 is in unknown state 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: mgrmap e35: smithi104.dpemzn(active, since 57s), standbys: smithi135.nwoumo 2024-04-27T01:11:32.833 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:32 smithi104 ceph-mon[63925]: from='mgr.14774 ' entity='' 2024-04-27T01:11:33.018 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:11:33.018 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: from='mgr.14774 172.21.15.104:0/3656118813' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: pgmap v32: 97 pgs: 97 active+clean; 330 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 730 op/s 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: mon.smithi104 calling monitor election 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: monmap epoch 2 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: last_changed 2024-04-27T01:00:59.463740+0000 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: created 2024-04-27T00:58:50.728785+0000 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: min_mon_release 17 (quincy) 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: election_strategy: 1 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:11:33.019 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: mgrmap e34: smithi104.dpemzn(active, since 57s), standbys: smithi135.nwoumo 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: daemon grafana.smithi104 on smithi104 is in unknown state 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: mgrmap e35: smithi104.dpemzn(active, since 57s), standbys: smithi135.nwoumo 2024-04-27T01:11:33.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:32 smithi135 ceph-mon[24080]: from='mgr.14774 ' entity='' 2024-04-27T01:11:33.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 148208 11.85 MB/sec execute 175 sec latency 28.743 ms 2024-04-27T01:11:34.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 148573 11.86 MB/sec execute 176 sec latency 38.822 ms 2024-04-27T01:11:34.857 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:34 smithi104 ceph-mon[63925]: mgrmap e36: smithi104.dpemzn(active, since 58s), standbys: smithi135.nwoumo 2024-04-27T01:11:35.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:34 smithi135 ceph-mon[24080]: mgrmap e36: smithi104.dpemzn(active, since 58s), standbys: smithi135.nwoumo 2024-04-27T01:11:35.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 148991 11.89 MB/sec execute 177 sec latency 30.757 ms 2024-04-27T01:11:36.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 149290 11.87 MB/sec execute 178 sec latency 47.254 ms 2024-04-27T01:11:37.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 149516 11.85 MB/sec execute 179 sec latency 45.921 ms 2024-04-27T01:11:38.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 149782 11.80 MB/sec execute 180 sec latency 56.304 ms 2024-04-27T01:11:39.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 150078 11.75 MB/sec execute 181 sec latency 49.160 ms 2024-04-27T01:11:40.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 150587 11.76 MB/sec execute 182 sec latency 39.110 ms 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:11:41.476 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:41 smithi104 ceph-mon[63925]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:11:41.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo restarted 2024-04-27T01:11:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: Standby manager daemon smithi135.nwoumo started 2024-04-27T01:11:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/crt"}]: dispatch 2024-04-27T01:11:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:11:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi135.nwoumo/key"}]: dispatch 2024-04-27T01:11:41.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:41 smithi135 ceph-mon[24080]: from='mgr.? 172.21.15.135:0/1685891073' entity='mgr.smithi135.nwoumo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:11:41.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 151261 11.78 MB/sec execute 183 sec latency 29.015 ms 2024-04-27T01:11:42.494 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:42 smithi104 ceph-mon[63925]: mgrmap e37: smithi104.dpemzn(active, since 65s), standbys: smithi135.nwoumo 2024-04-27T01:11:42.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:42 smithi135 ceph-mon[24080]: mgrmap e37: smithi104.dpemzn(active, since 65s), standbys: smithi135.nwoumo 2024-04-27T01:11:42.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 151723 11.78 MB/sec execute 184 sec latency 29.905 ms 2024-04-27T01:11:43.378 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: Activating manager daemon smithi104.dpemzn 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: mgrmap e38: smithi104.dpemzn(active, starting, since 0.00743847s), standbys: smithi135.nwoumo 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:11:43.505 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:11:43.506 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:11:43.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:11:43.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:43.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:43.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:11:43.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: Active manager daemon smithi104.dpemzn restarted 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: Activating manager daemon smithi104.dpemzn 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: mgrmap e38: smithi104.dpemzn(active, starting, since 0.00743847s), standbys: smithi135.nwoumo 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:11:43.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi104.dpemzn", "id": "smithi104.dpemzn"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr metadata", "who": "smithi135.nwoumo", "id": "smithi135.nwoumo"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:11:43.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: Manager daemon smithi104.dpemzn is now available 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:11:43.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:43 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.dpemzn/trash_purge_schedule"}]: dispatch 2024-04-27T01:11:43.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 152100 11.80 MB/sec execute 185 sec latency 55.656 ms 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: [27/Apr/2024:01:11:43] ENGINE Bus STARTING 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: mgrmap e39: smithi104.dpemzn(active, since 1.01531s), standbys: smithi135.nwoumo 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: [27/Apr/2024:01:11:43] ENGINE Serving on http://172.21.15.104:8765 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: [27/Apr/2024:01:11:43] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:11:44.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:44 smithi104 ceph-mon[63925]: [27/Apr/2024:01:11:43] ENGINE Bus STARTED 2024-04-27T01:11:44.556 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (2m) 23s ago 11m 24.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (11m) 23s ago 11m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (10m) 27s ago 10m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 unknown 23s ago 11m - - 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (5m) 23s ago 5m 39.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (5m) 27s ago 5m 16.2M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (90s) 23s ago 12m 495M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (32s) 27s ago 10m 109M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (12m) 23s ago 12m 51.5M 2048M 17.2.0 e1d6a67b021e 8177cde7a2ea 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (10m) 27s ago 10m 45.7M 2048M 17.2.0 e1d6a67b021e 1dbdd689cbb6 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (5m) 23s ago 5m 267M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:11:44.557 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (5m) 27s ago 5m 80.8M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (2m) 23s ago 11m 21.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (99s) 27s ago 10m 16.8M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (10m) 23s ago 10m 498M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (9m) 23s ago 9m 414M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (9m) 23s ago 9m 347M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (8m) 23s ago 8m 459M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (8m) 27s ago 8m 479M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (7m) 27s ago 7m 535M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (7m) 27s ago 7m 455M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (7m) 27s ago 7m 567M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:11:44.558 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (39s) 23s ago 10m 49.6M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: [27/Apr/2024:01:11:43] ENGINE Bus STARTING 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: mgrmap e39: smithi104.dpemzn(active, since 1.01531s), standbys: smithi135.nwoumo 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: [27/Apr/2024:01:11:43] ENGINE Serving on http://172.21.15.104:8765 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: pgmap v3: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: [27/Apr/2024:01:11:43] ENGINE Serving on https://172.21.15.104:7150 2024-04-27T01:11:44.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:44 smithi135 ceph-mon[24080]: [27/Apr/2024:01:11:43] ENGINE Bus STARTED 2024-04-27T01:11:44.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 152563 11.83 MB/sec execute 186 sec latency 37.279 ms 2024-04-27T01:11:45.254 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:45.288 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 3 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:11:45.289 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:11:45.552 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: from='client.24609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:45.552 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:11:45.552 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: Cluster is now healthy 2024-04-27T01:11:45.552 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:45.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: mgrmap e40: smithi104.dpemzn(active, since 2s), standbys: smithi135.nwoumo 2024-04-27T01:11:45.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: from='client.34126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:45.553 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:45 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2608161000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:45.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: from='client.24609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:45.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:11:45.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: Cluster is now healthy 2024-04-27T01:11:45.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: pgmap v4: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:45.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: mgrmap e40: smithi104.dpemzn(active, since 2s), standbys: smithi135.nwoumo 2024-04-27T01:11:45.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: from='client.34126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:45.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:45 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/2608161000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:45.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 152876 11.81 MB/sec execute 187 sec latency 46.894 ms 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:11:45.773 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "2/23 daemons upgraded", 2024-04-27T01:11:45.774 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:11:45.774 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:11:45.774 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:11:46.403 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:11:46.568 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:46 smithi104 ceph-mon[63925]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:46.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:46 smithi135 ceph-mon[24080]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:11:46.645 INFO:teuthology.orchestra.run.smithi104.stdout: 5 153148 11.79 MB/sec execute 188 sec latency 49.734 ms 2024-04-27T01:11:47.382 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:47 smithi104 ceph-mon[63925]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:47.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:47 smithi104 ceph-mon[63925]: mgrmap e41: smithi104.dpemzn(active, since 4s), standbys: smithi135.nwoumo 2024-04-27T01:11:47.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:47 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/502554597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:11:47.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:47.383 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:47.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 153466 11.75 MB/sec execute 189 sec latency 87.873 ms 2024-04-27T01:11:47.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:47 smithi135 ceph-mon[24080]: pgmap v5: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:47.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:47 smithi135 ceph-mon[24080]: mgrmap e41: smithi104.dpemzn(active, since 4s), standbys: smithi135.nwoumo 2024-04-27T01:11:47.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:47 smithi135 ceph-mon[24080]: from='client.? 172.21.15.104:0/502554597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:11:47.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:47 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:47.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:47 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:48.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 153860 11.72 MB/sec execute 190 sec latency 46.425 ms 2024-04-27T01:11:49.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 154344 11.75 MB/sec execute 191 sec latency 56.343 ms 2024-04-27T01:11:49.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:49 smithi104 ceph-mon[63925]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:49.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:49.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:49.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:49 smithi135 ceph-mon[24080]: pgmap v6: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-04-27T01:11:49.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:49 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:49.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:49 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:50.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 154865 11.74 MB/sec execute 192 sec latency 50.771 ms 2024-04-27T01:11:51.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 155220 11.71 MB/sec execute 193 sec latency 41.757 ms 2024-04-27T01:11:51.714 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:51 smithi104 ceph-mon[63925]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.0 MiB/s wr, 390 op/s 2024-04-27T01:11:51.715 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:51.715 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:51.901 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:51 smithi135 ceph-mon[24080]: pgmap v7: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.0 MiB/s wr, 390 op/s 2024-04-27T01:11:51.902 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:51 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:51.902 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:51 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:52.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 155505 11.72 MB/sec execute 194 sec latency 54.295 ms 2024-04-27T01:11:53.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 155859 11.72 MB/sec execute 195 sec latency 52.360 ms 2024-04-27T01:11:53.785 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:11:53.786 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:53 smithi104 ceph-mon[63925]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:11:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: pgmap v8: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-27T01:11:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-27T01:11:54.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:53 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.conf 2024-04-27T01:11:54.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 156298 11.75 MB/sec execute 196 sec latency 37.977 ms 2024-04-27T01:11:54.792 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:11:54.792 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:54.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T01:11:55.084 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:11:55.084 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:11:55.084 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:11:55.084 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: Updating smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:11:55.085 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:54 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-27T01:11:55.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 156715 11.75 MB/sec execute 197 sec latency 28.321 ms 2024-04-27T01:11:55.776 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:55 smithi104 ceph-mon[63925]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.3 MiB/s wr, 531 op/s 2024-04-27T01:11:55.776 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:55 smithi104 ceph-mon[63925]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:11:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:55 smithi135 ceph-mon[24080]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.3 MiB/s wr, 531 op/s 2024-04-27T01:11:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:55 smithi135 ceph-mon[24080]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:11:56.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 157204 11.71 MB/sec execute 198 sec latency 42.802 ms 2024-04-27T01:11:57.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: Upgrade: Updating mon.smithi135 2024-04-27T01:11:57.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:57.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:11:57.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:11:57.295 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:57.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: Deploying daemon mon.smithi135 on smithi135 2024-04-27T01:11:57.296 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:57 smithi104 ceph-mon[63925]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.9 MiB/s wr, 484 op/s 2024-04-27T01:11:57.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: Upgrade: Updating mon.smithi135 2024-04-27T01:11:57.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:11:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:11:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:11:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: Deploying daemon mon.smithi135 on smithi135 2024-04-27T01:11:57.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:57 smithi135 ceph-mon[24080]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.9 MiB/s wr, 484 op/s 2024-04-27T01:11:57.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 157671 11.72 MB/sec execute 199 sec latency 26.880 ms 2024-04-27T01:11:58.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:58 smithi135 ceph-mon[24080]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:58.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:11:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:58.525 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:58 smithi135 systemd[1]: Stopping Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:58.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 158259 11.73 MB/sec execute 200 sec latency 53.839 ms 2024-04-27T01:11:58.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135[24060]: 2024-04-27T01:11:58.535+0000 7f92427bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi135 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:11:58.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135[24060]: 2024-04-27T01:11:58.535+0000 7f92427bd700 -1 mon.smithi135@1(peon) e2 *** Got Signal Terminated *** 2024-04-27T01:11:58.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:58 smithi135 podman[49997]: 2024-04-27 01:11:58.616792932 +0000 UTC m=+0.204980111 container died 1dbdd689cbb6508229f85737668de1c5ee0e288856a3e9d23a6f220b92946858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.expose-services=, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, com.redhat.component=centos-stream-container) 2024-04-27T01:11:59.250 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 podman[49997]: 2024-04-27 01:11:59.042633096 +0000 UTC m=+0.630820277 container cleanup 1dbdd689cbb6508229f85737668de1c5ee0e288856a3e9d23a6f220b92946858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vendor=Red Hat, Inc., version=8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8) 2024-04-27T01:11:59.250 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 bash[49997]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135 2024-04-27T01:11:59.541 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 podman[50011]: 2024-04-27 01:11:59.261104565 +0000 UTC m=+0.638076380 container remove 1dbdd689cbb6508229f85737668de1c5ee0e288856a3e9d23a6f220b92946858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8) 2024-04-27T01:11:59.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 158842 11.74 MB/sec execute 201 sec latency 52.766 ms 2024-04-27T01:11:59.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi135.service: Deactivated successfully. 2024-04-27T01:11:59.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 systemd[1]: Stopped Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:59.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi135.service: Consumed 7.093s CPU time. 2024-04-27T01:11:59.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:11:59 smithi135 systemd[1]: Starting Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:12:00.282 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 podman[50140]: 2024-04-27 01:11:59.921959222 +0000 UTC m=+0.022408309 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:12:00.282 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 podman[50140]: 2024-04-27 01:12:00.029445773 +0000 UTC m=+0.129894911 container create 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 podman[50140]: 2024-04-27 01:12:00.29309555 +0000 UTC m=+0.393544638 container init 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 podman[50140]: 2024-04-27 01:12:00.29832701 +0000 UTC m=+0.398776083 container start 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD) 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: pidfile_write: ignore empty --pid-file 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: load: jerasure load: lrc 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Git sha 0 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T01:12:00.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: DB SUMMARY 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: DB Session ID: 4853OY26JNHJXL8163IF 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi135/store.db dir, Total Num: 1, files: 000024.sst 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi135/store.db: 000022.log size: 6610195 ; 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.env: 0x555c7b026c60 2024-04-27T01:12:00.616 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:12:00.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.info_log: 0x555c7d619960 2024-04-27T01:12:00.617 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.statistics: (nil) 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.use_fsync: 0 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:12:00.618 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.db_log_dir: 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.wal_dir: 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:12:00.619 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.write_buffer_manager: 0x555c7d5ec1e0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.unordered_write: 0 2024-04-27T01:12:00.620 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.row_cache: None 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.wal_filter: None 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:12:00.621 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.wal_compression: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:12:00.622 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_open_files: -1 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:12:00.623 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Compression algorithms supported: 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kZSTD supported: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:12:00.624 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000009 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.merge_operator: 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_filter: None 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555c7d619a60) 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:12:00.625 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_type: 0 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_index_type: 0 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_shortening: 1 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: checksum: 4 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: no_block_cache: 0 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache: 0x555c7d61d090 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_options: 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: capacity : 536870912 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: num_shard_bits : 4 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: strict_capacity_limit : 0 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_cache_compressed: (nil) 2024-04-27T01:12:00.626 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: persistent_cache: (nil) 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size: 4096 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_size_deviation: 10 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_restart_interval: 16 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: index_block_restart_interval: 1 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: metadata_block_size: 4096 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: partition_filters: 0 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: use_delta_encoding: 1 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: filter_policy: bloomfilter 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: whole_key_filtering: 1 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: verify_compression: 0 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: format_version: 5 2024-04-27T01:12:00.627 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_index_compression: 1 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: block_align: 0 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_auto_readahead_size: 262144 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: prepopulate_block_cache: 0 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression: NoCompression 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.num_levels: 7 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:12:00.628 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:12:00.629 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:12:00.630 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:12:00.631 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:12:00.632 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.ttl: 2592000 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:12:00.633 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi135/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12238, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a01204b4-f2a0-45b9-b505-deebd4aa034c 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180320337030, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180320354797, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4016762, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12243, "largest_seqno": 13935, "table_properties": {"data_size": 4009221, "index_size": 4794, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1733, "raw_key_size": 16177, "raw_average_key_size": 23, "raw_value_size": 3994279, "raw_average_value_size": 5814, "num_data_blocks": 218, "num_entries": 687, "num_filter_entries": 687, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714180320, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a01204b4-f2a0-45b9-b505-deebd4aa034c", "db_session_id": "4853OY26JNHJXL8163IF", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:12:00.634 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714180320354932, "job": 1, "event": "recovery_finished"} 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi135/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555c7d70e000 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: DB pointer 0x555c7d6f8000 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** DB Stats ** 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:12:00.635 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L0 1/0 3.83 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 483.1 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: L6 1/0 6.61 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Sum 2/0 10.44 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 483.1 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 483.1 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** Compaction Stats [default] ** 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:12:00.636 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 483.1 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Cumulative compaction: 0.00 GB write, 165.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Interval compaction: 0.00 GB write, 165.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:12:00.637 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Block cache BinnedLRUCache@0x555c7d61d090#2 capacity: 512.00 MB usage: 6.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: starting mon.smithi135 rank 1 at public addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] at bind addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi135 fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???) e2 preinit fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).mds e7 new map 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).mds e7 print_map 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: e7 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: legacy client fscid: 1 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Filesystem 'foofs' (1) 2024-04-27T01:12:00.638 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: fs_name foofs 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: epoch 5 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: created 2024-04-27T01:06:01.041077+0000 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: modified 2024-04-27T01:06:06.942013+0000 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: tableserver 0 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: root 0 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: session_timeout 60 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: session_autoclose 300 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_file_size 1099511627776 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_xattr_size 65536 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: required_client_features {} 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: last_failure 0 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: last_failure_osd_epoch 0 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-27T01:12:00.639 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: max_mds 1 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: in 0 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: up {0=14492} 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: failed 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: damaged 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: stopped 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: data_pools [3] 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: metadata_pool 2 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: inline_data disabled 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: balancer 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: bal_rank_mask -1 2024-04-27T01:12:00.640 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: standby_count_wanted 1 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: qdb_cluster leader: 0 members: 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.foofs.smithi104.wfnrbv{0:14492} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.104:6834/3832993091,v1:172.21.15.104:6835/3832993091] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: Standby daemons: 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout: [mds.foofs.smithi135.brxmmu{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] compat {c=[1],r=[1],i=[7ff]}] 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 ceph-mon[50158]: mon.smithi135@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 bash[50140]: 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 2024-04-27T01:12:00.641 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:00 smithi135 systemd[1]: Started Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:12:00.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 159286 11.77 MB/sec execute 202 sec latency 30.544 ms 2024-04-27T01:12:01.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 159761 11.79 MB/sec execute 203 sec latency 40.199 ms 2024-04-27T01:12:02.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 160048 11.79 MB/sec execute 204 sec latency 71.330 ms 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: mgrmap e41: smithi104.dpemzn(active, since 19s), standbys: smithi135.nwoumo 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: mon.smithi104 calling monitor election 2024-04-27T01:12:02.923 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: mon.smithi135 calling monitor election 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: monmap epoch 3 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: last_changed 2024-04-27T01:12:01.753782+0000 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: created 2024-04-27T00:58:50.728785+0000 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: min_mon_release 19 (squid) 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: election_strategy: 1 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: mgrmap e41: smithi104.dpemzn(active, since 19s), standbys: smithi135.nwoumo 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:02.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:02.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:02.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:02.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:02.925 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: mgrmap e41: smithi104.dpemzn(active, since 19s), standbys: smithi135.nwoumo 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:03.041 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: mon.smithi104 calling monitor election 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: mon.smithi135 calling monitor election 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: mon.smithi104 is new leader, mons smithi104,smithi135 in quorum (ranks 0,1) 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mon metadata", "id": "smithi135"}]: dispatch 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: monmap epoch 3 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: fsid 38f5b578-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: last_changed 2024-04-27T01:12:01.753782+0000 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: created 2024-04-27T00:58:50.728785+0000 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: min_mon_release 19 (squid) 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: election_strategy: 1 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: 0: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.smithi104 2024-04-27T01:12:03.042 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: 1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.smithi135 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: mgrmap e41: smithi104.dpemzn(active, since 19s), standbys: smithi135.nwoumo 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:03.043 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:03.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 160339 11.76 MB/sec execute 205 sec latency 46.053 ms 2024-04-27T01:12:03.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:03 smithi135 ceph-mon[50158]: pgmap v13: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 548 op/s 2024-04-27T01:12:03.963 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:03 smithi104 ceph-mon[63925]: pgmap v13: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 548 op/s 2024-04-27T01:12:04.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 160589 11.71 MB/sec execute 206 sec latency 77.691 ms 2024-04-27T01:12:05.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 160846 11.67 MB/sec execute 207 sec latency 58.836 ms 2024-04-27T01:12:06.008 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:05 smithi104 ceph-mon[63925]: pgmap v14: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 811 op/s 2024-04-27T01:12:06.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:05 smithi135 ceph-mon[50158]: pgmap v14: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 811 op/s 2024-04-27T01:12:06.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 161050 11.64 MB/sec execute 208 sec latency 67.096 ms 2024-04-27T01:12:07.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:07.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:07.023 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:06 smithi104 ceph-mon[63925]: pgmap v15: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-04-27T01:12:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:06 smithi135 ceph-mon[50158]: pgmap v15: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-04-27T01:12:07.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 161311 11.62 MB/sec execute 209 sec latency 70.270 ms 2024-04-27T01:12:08.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:08.289 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:08.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:08.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:08.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 161938 11.64 MB/sec execute 210 sec latency 58.233 ms 2024-04-27T01:12:09.348 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:09 smithi104 ceph-mon[63925]: pgmap v16: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-04-27T01:12:09.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:09 smithi135 ceph-mon[50158]: pgmap v16: 97 pgs: 97 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 546 op/s 2024-04-27T01:12:09.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 162452 11.65 MB/sec execute 211 sec latency 46.730 ms 2024-04-27T01:12:10.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 163003 11.70 MB/sec execute 212 sec latency 33.792 ms 2024-04-27T01:12:11.375 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: pgmap v17: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 743 op/s 2024-04-27T01:12:11.375 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:12:11.376 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:11.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: pgmap v17: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 743 op/s 2024-04-27T01:12:11.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:11.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:12:11.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:11.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:12:11.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:12:11.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:11.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 163581 11.74 MB/sec execute 213 sec latency 39.524 ms 2024-04-27T01:12:12.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:12 smithi104 ceph-mon[63925]: pgmap v18: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T01:12:12.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:12 smithi104 ceph-mon[63925]: pgmap v19: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-27T01:12:12.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:12 smithi104 ceph-mon[63925]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-27T01:12:12.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:12 smithi104 ceph-mon[63925]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-27T01:12:12.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:12 smithi135 ceph-mon[50158]: pgmap v18: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 500 op/s 2024-04-27T01:12:12.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:12 smithi135 ceph-mon[50158]: pgmap v19: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-04-27T01:12:12.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:12 smithi135 ceph-mon[50158]: Reconfiguring mon.smithi104 (monmap changed)... 2024-04-27T01:12:12.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:12 smithi135 ceph-mon[50158]: Reconfiguring daemon mon.smithi104 on smithi104 2024-04-27T01:12:12.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 164036 11.72 MB/sec execute 214 sec latency 39.011 ms 2024-04-27T01:12:13.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:13.646 INFO:teuthology.orchestra.run.smithi104.stdout: 5 164570 11.71 MB/sec execute 215 sec latency 57.758 ms 2024-04-27T01:12:14.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: pgmap v20: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 265 op/s 2024-04-27T01:12:14.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:14.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:14.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:12:14.428 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:12:14.429 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:14.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: pgmap v20: 97 pgs: 97 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 265 op/s 2024-04-27T01:12:14.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:14.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:14.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.dpemzn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:12:14.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:12:14.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:14.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 164903 11.69 MB/sec execute 216 sec latency 60.028 ms 2024-04-27T01:12:15.439 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:15 smithi104 ceph-mon[63925]: Reconfiguring mgr.smithi104.dpemzn (monmap changed)... 2024-04-27T01:12:15.439 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:15 smithi104 ceph-mon[63925]: Reconfiguring daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:12:15.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:15 smithi135 ceph-mon[50158]: Reconfiguring mgr.smithi104.dpemzn (monmap changed)... 2024-04-27T01:12:15.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:15 smithi135 ceph-mon[50158]: Reconfiguring daemon mgr.smithi104.dpemzn on smithi104 2024-04-27T01:12:15.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 165306 11.69 MB/sec execute 217 sec latency 44.219 ms 2024-04-27T01:12:16.453 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:16 smithi104 ceph-mon[63925]: pgmap v21: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-04-27T01:12:16.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:16 smithi135 ceph-mon[50158]: pgmap v21: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-04-27T01:12:16.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 165727 11.67 MB/sec execute 218 sec latency 58.466 ms 2024-04-27T01:12:16.912 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:12:17.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 166012 11.66 MB/sec execute 219 sec latency 47.046 ms 2024-04-27T01:12:17.870 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:12:17.870 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (3m) 26s ago 12m 24.1M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:12:17.870 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (12m) 26s ago 12m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:12:17.870 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (11m) 12s ago 11m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:12:17.870 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 26s ago 11m - - 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (6m) 26s ago 6m 39.8M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (6m) 12s ago 6m 16.6M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (2m) 26s ago 13m 479M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (65s) 12s ago 11m 448M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (46s) 26s ago 13m 33.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (17s) 12s ago 11m 26.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (5m) 26s ago 5m 267M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (5m) 12s ago 5m 81.1M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (2m) 26s ago 11m 21.4M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (2m) 12s ago 11m 17.1M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (10m) 26s ago 10m 543M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (10m) 26s ago 10m 465M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (9m) 26s ago 9m 380M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (9m) 26s ago 9m 488M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:12:17.871 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (8m) 12s ago 8m 525M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:12:17.872 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (8m) 12s ago 8m 322M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:12:17.872 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (8m) 12s ago 8m 524M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:12:17.872 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (7m) 12s ago 7m 394M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:12:17.872 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (72s) 26s ago 10m 59.6M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:12:17.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:17.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:17.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-27T01:12:17.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:17.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:17.963 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:17 smithi104 ceph-mon[63925]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: Reconfiguring crash.smithi104 (monmap changed)... 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:18.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:17 smithi135 ceph-mon[50158]: Reconfiguring daemon crash.smithi104 on smithi104 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:12:18.487 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:12:18.488 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:12:18.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 166400 11.68 MB/sec execute 220 sec latency 38.772 ms 2024-04-27T01:12:18.983 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:18 smithi104 ceph-mon[63925]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:18.983 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:18 smithi104 ceph-mon[63925]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-04-27T01:12:18.983 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:18 smithi104 ceph-mon[63925]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:18.983 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:18 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1520466193' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:12:19.074 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "4/23 daemons upgraded", 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading mon daemons", 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:12:19.075 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:12:19.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:18 smithi135 ceph-mon[50158]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:19.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:18 smithi135 ceph-mon[50158]: pgmap v22: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 690 op/s 2024-04-27T01:12:19.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:18 smithi135 ceph-mon[50158]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:19.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:18 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1520466193' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:19.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 166715 11.69 MB/sec execute 221 sec latency 58.058 ms 2024-04-27T01:12:19.713 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:19.713 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:19.713 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:19.976 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:19 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2573510030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:12:20.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:20.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:20.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:20.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:12:20.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:20.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:19 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2573510030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:12:20.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 167013 11.67 MB/sec execute 222 sec latency 52.380 ms 2024-04-27T01:12:20.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:20 smithi104 ceph-mon[63925]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-04-27T01:12:20.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:20 smithi104 ceph-mon[63925]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:20.758 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:20 smithi104 ceph-mon[63925]: Reconfiguring osd.0 (monmap changed)... 2024-04-27T01:12:20.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:20 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.0 on smithi104 2024-04-27T01:12:21.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:20 smithi135 ceph-mon[50158]: pgmap v23: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-04-27T01:12:21.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:20 smithi135 ceph-mon[50158]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:21.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:20 smithi135 ceph-mon[50158]: Reconfiguring osd.0 (monmap changed)... 2024-04-27T01:12:21.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:20 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.0 on smithi104 2024-04-27T01:12:21.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 167281 11.65 MB/sec execute 223 sec latency 56.631 ms 2024-04-27T01:12:22.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 167549 11.62 MB/sec execute 224 sec latency 59.186 ms 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: pgmap v24: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.8 MiB/s wr, 653 op/s 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: Reconfiguring osd.1 (monmap changed)... 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:12:22.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:22.797 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:22 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.1 on smithi104 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: pgmap v24: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.8 MiB/s wr, 653 op/s 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: Reconfiguring osd.1 (monmap changed)... 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:23.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:22 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.1 on smithi104 2024-04-27T01:12:23.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 167856 11.58 MB/sec execute 225 sec latency 48.598 ms 2024-04-27T01:12:24.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 168133 11.56 MB/sec execute 226 sec latency 51.813 ms 2024-04-27T01:12:24.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:24 smithi104 ceph-mon[63925]: pgmap v25: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-04-27T01:12:24.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:24.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:24.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:12:24.746 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:25.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:24 smithi135 ceph-mon[50158]: pgmap v25: 97 pgs: 97 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-04-27T01:12:25.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:25.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:25.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:12:25.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:25.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 168442 11.54 MB/sec execute 227 sec latency 38.856 ms 2024-04-27T01:12:25.798 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:25 smithi104 ceph-mon[63925]: Reconfiguring osd.2 (monmap changed)... 2024-04-27T01:12:25.798 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:25 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.2 on smithi104 2024-04-27T01:12:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:25 smithi135 ceph-mon[50158]: Reconfiguring osd.2 (monmap changed)... 2024-04-27T01:12:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:25 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.2 on smithi104 2024-04-27T01:12:26.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 168814 11.55 MB/sec execute 228 sec latency 45.691 ms 2024-04-27T01:12:26.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:26 smithi104 ceph-mon[63925]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 729 op/s 2024-04-27T01:12:27.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:26 smithi135 ceph-mon[50158]: pgmap v26: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 729 op/s 2024-04-27T01:12:27.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 169158 11.52 MB/sec execute 229 sec latency 52.079 ms 2024-04-27T01:12:27.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:27.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:27.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: Reconfiguring osd.3 (monmap changed)... 2024-04-27T01:12:27.836 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:12:27.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:27.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.3 on smithi104 2024-04-27T01:12:27.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: Reconfiguring osd.3 (monmap changed)... 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.3 on smithi104 2024-04-27T01:12:28.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:28.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 169508 11.51 MB/sec execute 230 sec latency 36.881 ms 2024-04-27T01:12:28.857 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:28 smithi104 ceph-mon[63925]: pgmap v27: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.2 MiB/s wr, 414 op/s 2024-04-27T01:12:29.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:28 smithi135 ceph-mon[50158]: pgmap v27: 97 pgs: 97 active+clean; 332 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.2 MiB/s wr, 414 op/s 2024-04-27T01:12:29.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 169749 11.51 MB/sec execute 231 sec latency 53.450 ms 2024-04-27T01:12:30.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 170100 11.52 MB/sec execute 232 sec latency 54.965 ms 2024-04-27T01:12:30.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:30 smithi104 ceph-mon[63925]: pgmap v28: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-27T01:12:30.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:30.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:30.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:12:30.896 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:30 smithi135 ceph-mon[50158]: pgmap v28: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 618 op/s 2024-04-27T01:12:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:12:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:31.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 170395 11.50 MB/sec execute 233 sec latency 55.261 ms 2024-04-27T01:12:31.942 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:31 smithi104 ceph-mon[63925]: Reconfiguring mds.foofs.smithi104.wfnrbv (monmap changed)... 2024-04-27T01:12:31.942 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:31 smithi104 ceph-mon[63925]: Reconfiguring daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:12:32.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:31 smithi135 ceph-mon[50158]: Reconfiguring mds.foofs.smithi104.wfnrbv (monmap changed)... 2024-04-27T01:12:32.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:31 smithi135 ceph-mon[50158]: Reconfiguring daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:12:32.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 170663 11.50 MB/sec execute 234 sec latency 56.764 ms 2024-04-27T01:12:32.965 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:32 smithi104 ceph-mon[63925]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-04-27T01:12:32.965 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:32.965 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:32.965 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:32.965 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:33.032 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:32 smithi135 ceph-mon[50158]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-04-27T01:12:33.032 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:33.032 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:33.032 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:33.032 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:33.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 170993 11.48 MB/sec execute 235 sec latency 39.114 ms 2024-04-27T01:12:33.982 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:33 smithi104 ceph-mon[63925]: Reconfiguring crash.smithi135 (monmap changed)... 2024-04-27T01:12:33.982 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:33 smithi104 ceph-mon[63925]: Reconfiguring daemon crash.smithi135 on smithi135 2024-04-27T01:12:34.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:33 smithi135 ceph-mon[50158]: Reconfiguring crash.smithi135 (monmap changed)... 2024-04-27T01:12:34.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:33 smithi135 ceph-mon[50158]: Reconfiguring daemon crash.smithi135 on smithi135 2024-04-27T01:12:34.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 171308 11.44 MB/sec execute 236 sec latency 47.296 ms 2024-04-27T01:12:35.001 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:34 smithi104 ceph-mon[63925]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-04-27T01:12:35.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:34 smithi135 ceph-mon[50158]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 388 op/s 2024-04-27T01:12:35.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 171652 11.43 MB/sec execute 237 sec latency 43.567 ms 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.9 MiB/s wr, 570 op/s 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: Reconfiguring mgr.smithi135.nwoumo (monmap changed)... 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:36.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:36 smithi104 ceph-mon[63925]: Reconfiguring daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.9 MiB/s wr, 570 op/s 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: Reconfiguring mgr.smithi135.nwoumo (monmap changed)... 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi135.nwoumo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:12:36.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:12:36.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:36.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:36 smithi135 ceph-mon[50158]: Reconfiguring daemon mgr.smithi135.nwoumo on smithi135 2024-04-27T01:12:36.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 171932 11.41 MB/sec execute 238 sec latency 43.862 ms 2024-04-27T01:12:37.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 172343 11.41 MB/sec execute 239 sec latency 45.985 ms 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: pgmap v32: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 385 op/s 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: Reconfiguring mon.smithi135 (monmap changed)... 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:12:38.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:12:38.319 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:38.319 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:38 smithi104 ceph-mon[63925]: Reconfiguring daemon mon.smithi135 on smithi135 2024-04-27T01:12:38.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: pgmap v32: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 385 op/s 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: Reconfiguring mon.smithi135 (monmap changed)... 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:38.615 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:38 smithi135 ceph-mon[50158]: Reconfiguring daemon mon.smithi135 on smithi135 2024-04-27T01:12:38.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 172706 11.39 MB/sec execute 240 sec latency 47.480 ms 2024-04-27T01:12:39.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 173096 11.39 MB/sec execute 241 sec latency 49.818 ms 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: pgmap v33: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 604 op/s 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: Reconfiguring osd.4 (monmap changed)... 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:40.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:40 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.4 on smithi135 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: pgmap v33: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 604 op/s 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: Reconfiguring osd.4 (monmap changed)... 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:40.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:40 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.4 on smithi135 2024-04-27T01:12:40.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 173484 11.41 MB/sec execute 242 sec latency 45.201 ms 2024-04-27T01:12:41.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 173816 11.42 MB/sec execute 243 sec latency 46.047 ms 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: pgmap v34: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 399 op/s 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: Reconfiguring osd.5 (monmap changed)... 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:42.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:42 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.5 on smithi135 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: pgmap v34: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 399 op/s 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: Reconfiguring osd.5 (monmap changed)... 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:42.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:42 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.5 on smithi135 2024-04-27T01:12:42.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 174236 11.42 MB/sec execute 244 sec latency 41.750 ms 2024-04-27T01:12:43.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:43.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:43.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:43.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:12:43.170 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:12:43.366 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:43.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 174666 11.41 MB/sec execute 245 sec latency 34.997 ms 2024-04-27T01:12:44.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:44 smithi135 ceph-mon[50158]: Reconfiguring osd.6 (monmap changed)... 2024-04-27T01:12:44.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:44 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.6 on smithi135 2024-04-27T01:12:44.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:44 smithi135 ceph-mon[50158]: pgmap v35: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 399 op/s 2024-04-27T01:12:44.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:44 smithi104 ceph-mon[63925]: Reconfiguring osd.6 (monmap changed)... 2024-04-27T01:12:44.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:44 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.6 on smithi135 2024-04-27T01:12:44.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:44 smithi104 ceph-mon[63925]: pgmap v35: 97 pgs: 97 active+clean; 323 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.0 MiB/s wr, 399 op/s 2024-04-27T01:12:44.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 175140 11.39 MB/sec execute 246 sec latency 35.967 ms 2024-04-27T01:12:45.647 INFO:teuthology.orchestra.run.smithi104.stdout: 5 175554 11.38 MB/sec execute 247 sec latency 45.199 ms 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: Reconfiguring osd.7 (monmap changed)... 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:45.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:45 smithi104 ceph-mon[63925]: Reconfiguring daemon osd.7 on smithi135 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: Reconfiguring osd.7 (monmap changed)... 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:46.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:45 smithi135 ceph-mon[50158]: Reconfiguring daemon osd.7 on smithi135 2024-04-27T01:12:46.654 INFO:teuthology.orchestra.run.smithi104.stdout: 5 176052 11.39 MB/sec execute 248 sec latency 40.613 ms 2024-04-27T01:12:46.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:46 smithi104 ceph-mon[63925]: pgmap v36: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.7 MiB/s wr, 628 op/s 2024-04-27T01:12:46.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:46.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:46.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:12:46.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:46 smithi135 ceph-mon[50158]: pgmap v36: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.7 MiB/s wr, 628 op/s 2024-04-27T01:12:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:12:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:47.654 INFO:teuthology.orchestra.run.smithi104.stdout: 5 176551 11.40 MB/sec execute 249 sec latency 31.293 ms 2024-04-27T01:12:47.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:47 smithi104 ceph-mon[63925]: Reconfiguring mds.foofs.smithi135.brxmmu (monmap changed)... 2024-04-27T01:12:47.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:47 smithi104 ceph-mon[63925]: Reconfiguring daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:12:47.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:47 smithi135 ceph-mon[50158]: Reconfiguring mds.foofs.smithi135.brxmmu (monmap changed)... 2024-04-27T01:12:47.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:47 smithi135 ceph-mon[50158]: Reconfiguring daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:12:48.654 INFO:teuthology.orchestra.run.smithi104.stdout: 5 177039 11.43 MB/sec execute 250 sec latency 50.480 ms 2024-04-27T01:12:48.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: pgmap v37: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-27T01:12:48.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:48.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]: dispatch 2024-04-27T01:12:48.756 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]': finished 2024-04-27T01:12:48.757 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]: dispatch 2024-04-27T01:12:48.757 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]': finished 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: pgmap v37: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]: dispatch 2024-04-27T01:12:49.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi104"}]': finished 2024-04-27T01:12:49.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]: dispatch 2024-04-27T01:12:49.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi135"}]': finished 2024-04-27T01:12:49.654 INFO:teuthology.orchestra.run.smithi104.stdout: 5 177456 11.44 MB/sec execute 251 sec latency 38.653 ms 2024-04-27T01:12:49.767 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:49 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all mon 2024-04-27T01:12:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:49 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all mon 2024-04-27T01:12:50.280 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:12:50.654 INFO:teuthology.orchestra.run.smithi104.stdout: 5 177817 11.44 MB/sec execute 252 sec latency 71.402 ms 2024-04-27T01:12:50.774 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:50 smithi104 ceph-mon[63925]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 735 op/s 2024-04-27T01:12:51.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:50 smithi135 ceph-mon[50158]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 735 op/s 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (3m) 60s ago 12m 24.1M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (12m) 60s ago 12m 6405k - 17.2.0 e1d6a67b021e a3b2be54dbce 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (12m) 45s ago 12m 6400k - 17.2.0 e1d6a67b021e 35e0582e669b 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 60s ago 12m - - 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (6m) 60s ago 6m 39.8M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (6m) 45s ago 6m 16.6M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (2m) 60s ago 13m 479M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:12:51.354 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (99s) 45s ago 11m 448M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (79s) 60s ago 13m 33.2M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (51s) 45s ago 11m 26.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (6m) 60s ago 6m 267M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (6m) 45s ago 6m 81.1M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (3m) 60s ago 12m 21.4M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (2m) 45s ago 11m 17.1M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (11m) 60s ago 11m 543M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (10m) 60s ago 10m 465M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (10m) 60s ago 10m 380M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (9m) 60s ago 9m 488M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (9m) 45s ago 9m 525M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (9m) 45s ago 9m 322M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (8m) 45s ago 8m 524M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (8m) 45s ago 8m 394M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:12:51.355 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (105s) 60s ago 11m 59.6M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:12:51.655 INFO:teuthology.orchestra.run.smithi104.stdout: 5 178145 11.42 MB/sec execute 253 sec latency 73.500 ms 2024-04-27T01:12:51.798 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:51 smithi104 ceph-mon[63925]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:51.798 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:51 smithi104 ceph-mon[63925]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:52.011 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:12:52.012 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:12:52.013 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T01:12:52.013 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T01:12:52.013 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:12:52.013 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:12:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:51 smithi135 ceph-mon[50158]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:51 smithi135 ceph-mon[50158]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:52.583 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:12:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:12:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:12:52.583 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "4/23 daemons upgraded", 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading crash daemons", 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:12:52.584 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:12:52.655 INFO:teuthology.orchestra.run.smithi104.stdout: 5 178462 11.39 MB/sec execute 254 sec latency 61.872 ms 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: pgmap v39: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 516 op/s 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3870823757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:52.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:52 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: pgmap v39: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 516 op/s 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3870823757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:12:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:52 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:12:53.255 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:12:53.255 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:12:53.255 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:12:53.655 INFO:teuthology.orchestra.run.smithi104.stdout: 5 178823 11.37 MB/sec execute 255 sec latency 38.147 ms 2024-04-27T01:12:53.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:53 smithi104 ceph-mon[63925]: Upgrade: Updating crash.smithi104 (1/2) 2024-04-27T01:12:53.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:53 smithi104 ceph-mon[63925]: Deploying daemon crash.smithi104 on smithi104 2024-04-27T01:12:53.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:53 smithi104 ceph-mon[63925]: from='client.44139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:53.837 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:53 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3178496227' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:12:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:53 smithi135 ceph-mon[50158]: Upgrade: Updating crash.smithi104 (1/2) 2024-04-27T01:12:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:53 smithi135 ceph-mon[50158]: Deploying daemon crash.smithi104 on smithi104 2024-04-27T01:12:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:53 smithi135 ceph-mon[50158]: from='client.44139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:12:54.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:53 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3178496227' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:12:54.655 INFO:teuthology.orchestra.run.smithi104.stdout: 5 179232 11.38 MB/sec execute 256 sec latency 62.090 ms 2024-04-27T01:12:54.755 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:54 smithi104 ceph-mon[63925]: pgmap v40: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 516 op/s 2024-04-27T01:12:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:54 smithi135 ceph-mon[50158]: pgmap v40: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 516 op/s 2024-04-27T01:12:55.657 INFO:teuthology.orchestra.run.smithi104.stdout: 5 179806 11.39 MB/sec execute 257 sec latency 51.159 ms 2024-04-27T01:12:56.657 INFO:teuthology.orchestra.run.smithi104.stdout: 5 180486 11.44 MB/sec execute 258 sec latency 32.666 ms 2024-04-27T01:12:56.764 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:56 smithi104 ceph-mon[63925]: pgmap v41: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 743 op/s 2024-04-27T01:12:57.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:56 smithi135 ceph-mon[50158]: pgmap v41: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 743 op/s 2024-04-27T01:12:57.661 INFO:teuthology.orchestra.run.smithi104.stdout: 5 181157 11.47 MB/sec execute 259 sec latency 22.489 ms 2024-04-27T01:12:57.768 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:58.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:58.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 181734 11.49 MB/sec execute 260 sec latency 34.435 ms 2024-04-27T01:12:58.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:58 smithi104 ceph-mon[63925]: pgmap v42: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 514 op/s 2024-04-27T01:12:59.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:58 smithi135 ceph-mon[50158]: pgmap v42: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.0 MiB/s wr, 514 op/s 2024-04-27T01:12:59.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 182041 11.46 MB/sec execute 261 sec latency 84.769 ms 2024-04-27T01:12:59.826 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:12:59.826 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:12:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:00.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:00.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:12:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:00.661 INFO:teuthology.orchestra.run.smithi104.stdout: 5 182331 11.43 MB/sec execute 262 sec latency 79.747 ms 2024-04-27T01:13:00.853 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:00 smithi104 ceph-mon[63925]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 859 op/s 2024-04-27T01:13:01.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:00 smithi135 ceph-mon[50158]: pgmap v43: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 859 op/s 2024-04-27T01:13:01.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 182604 11.41 MB/sec execute 263 sec latency 47.633 ms 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 570 op/s 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: Upgrade: Updating crash.smithi135 (2/2) 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:02.373 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:02 smithi104 ceph-mon[63925]: Deploying daemon crash.smithi135 on smithi135 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: pgmap v44: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 570 op/s 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: Upgrade: Updating crash.smithi135 (2/2) 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi135", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:02.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:02 smithi135 ceph-mon[50158]: Deploying daemon crash.smithi135 on smithi135 2024-04-27T01:13:02.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 183265 11.44 MB/sec execute 264 sec latency 54.756 ms 2024-04-27T01:13:03.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 183963 11.48 MB/sec execute 265 sec latency 19.253 ms 2024-04-27T01:13:04.384 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:04 smithi104 ceph-mon[63925]: pgmap v45: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 570 op/s 2024-04-27T01:13:04.583 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:04 smithi135 ceph-mon[50158]: pgmap v45: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.0 MiB/s wr, 570 op/s 2024-04-27T01:13:04.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 184409 11.49 MB/sec execute 266 sec latency 40.917 ms 2024-04-27T01:13:05.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 184751 11.49 MB/sec execute 267 sec latency 41.642 ms 2024-04-27T01:13:06.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:06 smithi104 ceph-mon[63925]: pgmap v46: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 834 op/s 2024-04-27T01:13:06.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:06.430 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:06.431 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:06.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 184962 11.47 MB/sec execute 268 sec latency 65.328 ms 2024-04-27T01:13:06.761 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:06 smithi135 ceph-mon[50158]: pgmap v46: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 834 op/s 2024-04-27T01:13:06.761 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:06.761 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:06.761 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:07.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 185328 11.46 MB/sec execute 269 sec latency 40.440 ms 2024-04-27T01:13:08.465 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:08 smithi104 ceph-mon[63925]: pgmap v47: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 608 op/s 2024-04-27T01:13:08.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 185900 11.45 MB/sec execute 270 sec latency 33.207 ms 2024-04-27T01:13:08.784 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:08 smithi135 ceph-mon[50158]: pgmap v47: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 608 op/s 2024-04-27T01:13:09.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 186415 11.46 MB/sec execute 271 sec latency 41.974 ms 2024-04-27T01:13:10.479 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:10 smithi104 ceph-mon[63925]: pgmap v48: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 849 op/s 2024-04-27T01:13:10.479 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:10.479 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 186940 11.47 MB/sec execute 272 sec latency 37.034 ms 2024-04-27T01:13:10.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:10 smithi135 ceph-mon[50158]: pgmap v48: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 849 op/s 2024-04-27T01:13:10.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:10.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:11.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 187374 11.47 MB/sec execute 273 sec latency 68.642 ms 2024-04-27T01:13:12.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 187820 11.49 MB/sec execute 274 sec latency 38.239 ms 2024-04-27T01:13:12.765 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:12 smithi104 ceph-mon[63925]: pgmap v49: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-04-27T01:13:12.765 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:12.765 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:12.765 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:12.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:12 smithi135 ceph-mon[50158]: pgmap v49: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-04-27T01:13:12.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:12.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:12.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:13.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 188270 11.50 MB/sec execute 275 sec latency 34.818 ms 2024-04-27T01:13:14.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 188563 11.49 MB/sec execute 276 sec latency 50.462 ms 2024-04-27T01:13:14.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:14 smithi104 ceph-mon[63925]: pgmap v50: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-04-27T01:13:14.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:14.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:14.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:14 smithi135 ceph-mon[50158]: pgmap v50: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-04-27T01:13:14.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:14.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:15.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 188797 11.47 MB/sec execute 277 sec latency 76.113 ms 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 797 op/s 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: mgrmap e42: smithi104.dpemzn(active, since 92s), standbys: smithi135.nwoumo 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 797 op/s 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: mgrmap e42: smithi104.dpemzn(active, since 92s), standbys: smithi135.nwoumo 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:16.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 189067 11.44 MB/sec execute 278 sec latency 72.815 ms 2024-04-27T01:13:17.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 189316 11.42 MB/sec execute 279 sec latency 54.496 ms 2024-04-27T01:13:18.113 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:18 smithi104 ceph-mon[63925]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 533 op/s 2024-04-27T01:13:18.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:18 smithi135 ceph-mon[50158]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 533 op/s 2024-04-27T01:13:18.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 189603 11.40 MB/sec execute 280 sec latency 41.255 ms 2024-04-27T01:13:19.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 189904 11.39 MB/sec execute 281 sec latency 55.523 ms 2024-04-27T01:13:20.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:20 smithi104 ceph-mon[63925]: pgmap v53: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 713 op/s 2024-04-27T01:13:20.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:20 smithi135 ceph-mon[50158]: pgmap v53: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 713 op/s 2024-04-27T01:13:20.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 190278 11.38 MB/sec execute 282 sec latency 55.798 ms 2024-04-27T01:13:21.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 190587 11.37 MB/sec execute 283 sec latency 36.184 ms 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: pgmap v54: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: pgmap v55: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.6 MiB/s wr, 559 op/s 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: pgmap v56: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-27T01:13:22.179 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all crash 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]': finished 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]': finished 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T01:13:22.180 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:22 smithi104 ceph-mon[63925]: Upgrade: osd.0 is safe to restart 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: pgmap v54: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: pgmap v55: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.6 MiB/s wr, 559 op/s 2024-04-27T01:13:22.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: pgmap v56: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 697 op/s 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all crash 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi104"}]': finished 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi135"}]': finished 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-27T01:13:22.365 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:22 smithi135 ceph-mon[50158]: Upgrade: osd.0 is safe to restart 2024-04-27T01:13:22.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 190848 11.36 MB/sec execute 284 sec latency 68.828 ms 2024-04-27T01:13:23.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 191279 11.37 MB/sec execute 285 sec latency 30.767 ms 2024-04-27T01:13:23.829 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:13:24.199 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:24 smithi104 ceph-mon[63925]: pgmap v57: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.4 MiB/s wr, 266 op/s 2024-04-27T01:13:24.200 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:24 smithi104 ceph-mon[63925]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:24.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:24 smithi135 ceph-mon[50158]: pgmap v57: 97 pgs: 97 active+clean; 327 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.4 MiB/s wr, 266 op/s 2024-04-27T01:13:24.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:24 smithi135 ceph-mon[50158]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:24.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 191623 11.37 MB/sec execute 286 sec latency 38.263 ms 2024-04-27T01:13:24.949 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (4m) 10s ago 13m 24.1M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (27s) 10s ago 13m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (20s) 14s ago 12m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 10s ago 12m - - 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (7m) 10s ago 7m 40.7M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (7m) 14s ago 7m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (3m) 10s ago 14m 500M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (2m) 14s ago 12m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (113s) 10s ago 14m 44.8M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (85s) 14s ago 12m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (6m) 10s ago 6m 265M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:13:24.950 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (6m) 14s ago 6m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (3m) 10s ago 12m 18.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (3m) 14s ago 12m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (11m) 10s ago 11m 387M 1936M 17.2.0 e1d6a67b021e 95ab76383865 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (11m) 10s ago 11m 568M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (10m) 10s ago 10m 481M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (10m) 10s ago 10m 574M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (9m) 14s ago 9m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (9m) 14s ago 9m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (9m) 14s ago 9m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (8m) 14s ago 8m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:13:24.951 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (2m) 10s ago 12m 66.1M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:13:25.221 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: Upgrade: Updating osd.0 2024-04-27T01:13:25.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:25.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:13:25.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:25.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: Deploying daemon osd.0 on smithi104 2024-04-27T01:13:25.222 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:25 smithi104 ceph-mon[63925]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: Upgrade: Updating osd.0 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: Deploying daemon osd.0 on smithi104 2024-04-27T01:13:25.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:25 smithi135 ceph-mon[50158]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:25.619 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:13:25.620 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-27T01:13:25.620 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T01:13:25.620 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:13:25.620 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:13:25.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 191909 11.36 MB/sec execute 287 sec latency 59.640 ms 2024-04-27T01:13:26.081 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "6/23 daemons upgraded", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:13:26.082 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:13:26.251 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:26 smithi104 ceph-mon[63925]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:26.251 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:26 smithi104 ceph-mon[63925]: pgmap v58: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-04-27T01:13:26.251 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:26 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1581700219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:26.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:26 smithi135 ceph-mon[50158]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:26.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:26 smithi135 ceph-mon[50158]: pgmap v58: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-04-27T01:13:26.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:26 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1581700219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:26.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 192126 11.34 MB/sec execute 288 sec latency 66.334 ms 2024-04-27T01:13:26.681 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:13:26.681 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:13:26.682 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:13:27.261 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:27 smithi104 ceph-mon[63925]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:27.261 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:27 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1064040903' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:13:27.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:27 smithi135 ceph-mon[50158]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:27.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:27 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1064040903' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:13:27.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 192382 11.33 MB/sec execute 289 sec latency 63.753 ms 2024-04-27T01:13:28.279 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:28 smithi104 ceph-mon[63925]: pgmap v59: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.0 MiB/s wr, 290 op/s 2024-04-27T01:13:28.279 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:28.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:28 smithi135 ceph-mon[50158]: pgmap v59: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.0 MiB/s wr, 290 op/s 2024-04-27T01:13:28.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:28.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 192710 11.30 MB/sec execute 290 sec latency 60.602 ms 2024-04-27T01:13:29.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 193092 11.28 MB/sec execute 291 sec latency 45.419 ms 2024-04-27T01:13:30.316 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:30 smithi104 ceph-mon[63925]: pgmap v60: 97 pgs: 97 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 560 op/s 2024-04-27T01:13:30.316 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:30 smithi104 systemd[1]: Stopping Ceph osd.0 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:13:30.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:30 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[31952]: 2024-04-27T01:13:30.502+0000 7f528c061700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:13:30.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:30 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[31952]: 2024-04-27T01:13:30.502+0000 7f528c061700 -1 osd.0 76 *** Got signal Terminated *** 2024-04-27T01:13:30.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:30 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[31952]: 2024-04-27T01:13:30.502+0000 7f528c061700 -1 osd.0 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:13:30.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:30 smithi135 ceph-mon[50158]: pgmap v60: 97 pgs: 97 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 560 op/s 2024-04-27T01:13:30.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 193508 11.29 MB/sec execute 292 sec latency 51.308 ms 2024-04-27T01:13:31.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:31 smithi104 ceph-mon[63925]: osd.0 marked itself down and dead 2024-04-27T01:13:31.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:31 smithi135 ceph-mon[50158]: osd.0 marked itself down and dead 2024-04-27T01:13:31.662 INFO:teuthology.orchestra.run.smithi104.stdout: 5 193888 11.29 MB/sec execute 293 sec latency 412.328 ms 2024-04-27T01:13:32.164 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:31 smithi104 podman[69496]: 2024-04-27 01:13:31.894999571 +0000 UTC m=+1.594402491 container died 95ab76383865c7b8d2ead7d940b94cfc662a8c431af380a4e0d92552d1f9aefe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.expose-services=, name=centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD) 2024-04-27T01:13:32.455 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:32 smithi104 ceph-mon[63925]: pgmap v61: 97 pgs: 97 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 455 op/s 2024-04-27T01:13:32.456 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:32 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:13:32.456 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:32 smithi104 ceph-mon[63925]: osdmap e77: 8 total, 7 up, 8 in 2024-04-27T01:13:32.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:32 smithi135 ceph-mon[50158]: pgmap v61: 97 pgs: 97 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 455 op/s 2024-04-27T01:13:32.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:32 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:13:32.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:32 smithi135 ceph-mon[50158]: osdmap e77: 8 total, 7 up, 8 in 2024-04-27T01:13:32.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 194136 11.27 MB/sec execute 294 sec latency 989.870 ms 2024-04-27T01:13:32.961 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:32 smithi104 podman[69496]: 2024-04-27 01:13:32.841267793 +0000 UTC m=+2.540670708 container cleanup 95ab76383865c7b8d2ead7d940b94cfc662a8c431af380a4e0d92552d1f9aefe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, architecture=x86_64, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, name=centos-stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, version=8, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-27T01:13:32.962 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:32 smithi104 bash[69496]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0 2024-04-27T01:13:33.465 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:33 smithi104 ceph-mon[63925]: osdmap e78: 8 total, 7 up, 8 in 2024-04-27T01:13:33.465 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:33 smithi104 podman[69513]: 2024-04-27 01:13:33.292248113 +0000 UTC m=+1.400693033 container remove 95ab76383865c7b8d2ead7d940b94cfc662a8c431af380a4e0d92552d1f9aefe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, RELEASE=HEAD) 2024-04-27T01:13:33.515 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:33 smithi135 ceph-mon[50158]: osdmap e78: 8 total, 7 up, 8 in 2024-04-27T01:13:33.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 194558 11.28 MB/sec execute 295 sec latency 32.810 ms 2024-04-27T01:13:33.969 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:33 smithi104 podman[69601]: 2024-04-27 01:13:33.672625138 +0000 UTC m=+0.024409147 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:13:33.969 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:33 smithi104 podman[69601]: 2024-04-27 01:13:33.808150992 +0000 UTC m=+0.159935040 container create e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:13:34.226 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:34 smithi104 ceph-mon[63925]: pgmap v64: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-04-27T01:13:34.226 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:34 smithi104 podman[69601]: 2024-04-27 01:13:34.164194821 +0000 UTC m=+0.515978837 container init e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-27T01:13:34.226 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:34 smithi104 podman[69601]: 2024-04-27 01:13:34.174091653 +0000 UTC m=+0.525875657 container start e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T01:13:34.476 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:34 smithi104 podman[69601]: 2024-04-27 01:13:34.291643849 +0000 UTC m=+0.643427856 container attach e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-27T01:13:34.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:34 smithi135 ceph-mon[50158]: pgmap v64: 97 pgs: 13 stale+active+clean, 84 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.7 MiB/s wr, 274 op/s 2024-04-27T01:13:34.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 194995 11.29 MB/sec execute 296 sec latency 32.284 ms 2024-04-27T01:13:34.731 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:34 smithi104 podman[69601]: 2024-04-27 01:13:34.623423537 +0000 UTC m=+0.975207542 container died e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-27T01:13:35.489 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:35 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 215/1605 objects degraded (13.396%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:35.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:35 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 215/1605 objects degraded (13.396%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:35.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 195447 11.30 MB/sec execute 297 sec latency 52.439 ms 2024-04-27T01:13:35.995 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:35 smithi104 podman[69643]: 2024-04-27 01:13:35.853032914 +0000 UTC m=+1.218406749 container remove e31dabeccd8197cc53e88ef099dffd271a43c816c98ac57e2b3bd871e9f77485 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-27T01:13:35.996 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:35 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-27T01:13:35.996 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:35 smithi104 systemd[1]: Stopped Ceph osd.0 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:13:35.996 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:35 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.0.service: Consumed 3min 18.950s CPU time. 2024-04-27T01:13:36.246 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:36 smithi104 systemd[1]: Starting Ceph osd.0 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:13:36.406 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:36 smithi104 ceph-mon[63925]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 328 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 610 op/s; 215/1605 objects degraded (13.396%) 2024-04-27T01:13:36.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:36 smithi135 ceph-mon[50158]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 328 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 610 op/s; 215/1605 objects degraded (13.396%) 2024-04-27T01:13:36.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 195882 11.30 MB/sec execute 298 sec latency 29.391 ms 2024-04-27T01:13:36.751 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:36 smithi104 podman[69751]: 2024-04-27 01:13:36.445477641 +0000 UTC m=+0.046016425 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:13:36.752 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:36 smithi104 podman[69751]: 2024-04-27 01:13:36.602781329 +0000 UTC m=+0.203320089 container create 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-27T01:13:37.004 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:36 smithi104 podman[69751]: 2024-04-27 01:13:36.969760019 +0000 UTC m=+0.570298779 container init 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T01:13:37.004 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:36 smithi104 podman[69751]: 2024-04-27 01:13:36.975673548 +0000 UTC m=+0.576212313 container start 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb) 2024-04-27T01:13:37.255 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 podman[69751]: 2024-04-27 01:13:37.123531744 +0000 UTC m=+0.724070514 container attach 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T01:13:37.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 196302 11.28 MB/sec execute 299 sec latency 40.197 ms 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T01:13:37.775 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 bash[69751]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate[69764]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-27T01:13:37.776 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:37 smithi104 podman[69751]: 2024-04-27 01:13:37.650724506 +0000 UTC m=+1.251263276 container died 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-27T01:13:38.541 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:38 smithi104 ceph-mon[63925]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 328 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.2 MiB/s wr, 335 op/s; 215/1605 objects degraded (13.396%) 2024-04-27T01:13:38.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:38 smithi135 ceph-mon[50158]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 328 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.2 MiB/s wr, 335 op/s; 215/1605 objects degraded (13.396%) 2024-04-27T01:13:38.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 196667 11.27 MB/sec execute 300 sec latency 35.666 ms 2024-04-27T01:13:39.056 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:38 smithi104 podman[69895]: 2024-04-27 01:13:38.828549454 +0000 UTC m=+1.168508710 container remove 5378e04dba2dbca1665e31455c8f99c953ee3656c0bccc120db308d25542a634 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-activate, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T01:13:39.057 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 podman[69959]: 2024-04-27 01:13:38.962772558 +0000 UTC m=+0.021503520 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:13:39.308 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 podman[69959]: 2024-04-27 01:13:39.10761553 +0000 UTC m=+0.166346482 container create 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T01:13:39.559 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 podman[69959]: 2024-04-27 01:13:39.449995208 +0000 UTC m=+0.508726152 container init 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb) 2024-04-27T01:13:39.560 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 podman[69959]: 2024-04-27 01:13:39.454848179 +0000 UTC m=+0.513579127 container start 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-27T01:13:39.560 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 ceph-osd[69979]: -- 172.21.15.104:0/1356270758 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55afdab6b680 con 0x55afdab35000 2024-04-27T01:13:39.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 197021 11.27 MB/sec execute 301 sec latency 43.419 ms 2024-04-27T01:13:39.810 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 bash[69959]: 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c 2024-04-27T01:13:39.811 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:39 smithi104 systemd[1]: Started Ceph osd.0 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:13:40.066 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:40 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:13:40.037+0000 7f646ea227c0 -1 Falling back to public interface 2024-04-27T01:13:40.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 197443 11.27 MB/sec execute 302 sec latency 47.832 ms 2024-04-27T01:13:40.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:40 smithi104 ceph-mon[63925]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.2 MiB/s wr, 729 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:40.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:40.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:40.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:41.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:40 smithi135 ceph-mon[50158]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.2 MiB/s wr, 729 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:41.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:41.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:41.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:41.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 197854 11.27 MB/sec execute 303 sec latency 53.615 ms 2024-04-27T01:13:42.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 198249 11.27 MB/sec execute 304 sec latency 33.547 ms 2024-04-27T01:13:42.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:42 smithi104 ceph-mon[63925]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.8 MiB/s wr, 588 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:42.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:42 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 227/1695 objects degraded (13.392%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:42.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:42.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:43.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:42 smithi135 ceph-mon[50158]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.8 MiB/s wr, 588 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:43.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:42 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 227/1695 objects degraded (13.392%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:43.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:43.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:43.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 198525 11.27 MB/sec execute 305 sec latency 39.857 ms 2024-04-27T01:13:44.663 INFO:teuthology.orchestra.run.smithi104.stdout: 5 198813 11.26 MB/sec execute 306 sec latency 45.977 ms 2024-04-27T01:13:44.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:44 smithi104 ceph-mon[63925]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:45.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:44 smithi135 ceph-mon[50158]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 534 op/s; 227/1695 objects degraded (13.392%) 2024-04-27T01:13:45.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 199060 11.25 MB/sec execute 307 sec latency 53.918 ms 2024-04-27T01:13:46.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 199288 11.24 MB/sec execute 308 sec latency 83.099 ms 2024-04-27T01:13:46.740 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:46 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:13:46.614+0000 7f646ea227c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-27T01:13:46.995 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:46 smithi104 ceph-mon[63925]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 330 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.2 MiB/s wr, 713 op/s; 230/1662 objects degraded (13.839%) 2024-04-27T01:13:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:46 smithi135 ceph-mon[50158]: pgmap v70: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 330 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.2 MiB/s wr, 713 op/s; 230/1662 objects degraded (13.839%) 2024-04-27T01:13:47.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 199545 11.21 MB/sec execute 309 sec latency 63.116 ms 2024-04-27T01:13:47.771 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:47 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 230/1662 objects degraded (13.839%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:48.029 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:47 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:13:47.870+0000 7f646ea227c0 -1 osd.0 76 log_to_monitors true 2024-04-27T01:13:48.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:47 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 230/1662 objects degraded (13.839%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:48.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 199951 11.20 MB/sec execute 310 sec latency 41.359 ms 2024-04-27T01:13:48.788 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:48 smithi104 ceph-mon[63925]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 330 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 489 op/s; 230/1662 objects degraded (13.839%) 2024-04-27T01:13:48.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:48.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:48.789 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:48 smithi104 ceph-mon[63925]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:13:49.039 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:13:48 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:13:48.799+0000 7f645fe7a700 -1 osd.0 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:13:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:48 smithi135 ceph-mon[50158]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 330 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.1 MiB/s wr, 489 op/s; 230/1662 objects degraded (13.839%) 2024-04-27T01:13:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:49.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:48 smithi135 ceph-mon[50158]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:13:49.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 200345 11.19 MB/sec execute 311 sec latency 38.844 ms 2024-04-27T01:13:49.800 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:49 smithi104 ceph-mon[63925]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:13:49.801 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:49 smithi104 ceph-mon[63925]: osdmap e79: 8 total, 7 up, 8 in 2024-04-27T01:13:49.801 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:49 smithi104 ceph-mon[63925]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:13:49.801 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:49.801 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:49 smithi135 ceph-mon[50158]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:13:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:49 smithi135 ceph-mon[50158]: osdmap e79: 8 total, 7 up, 8 in 2024-04-27T01:13:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:49 smithi135 ceph-mon[50158]: from='osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:13:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:50.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 200446 11.16 MB/sec execute 312 sec latency 854.054 ms 2024-04-27T01:13:50.958 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:50 smithi104 ceph-mon[63925]: pgmap v73: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:50.958 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:50 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:13:50.958 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:50 smithi104 ceph-mon[63925]: osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041] boot 2024-04-27T01:13:50.958 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:50 smithi104 ceph-mon[63925]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T01:13:50.958 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:13:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:50 smithi135 ceph-mon[50158]: pgmap v73: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 488 op/s; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:51.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:50 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:13:51.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:50 smithi135 ceph-mon[50158]: osd.0 [v2:172.21.15.104:6802/1130089041,v1:172.21.15.104:6803/1130089041] boot 2024-04-27T01:13:51.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:50 smithi135 ceph-mon[50158]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T01:13:51.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:13:51.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 200553 11.14 MB/sec execute 313 sec latency 1784.026 ms 2024-04-27T01:13:52.072 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:51 smithi104 ceph-mon[63925]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T01:13:52.073 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:51 smithi104 ceph-mon[63925]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T01:13:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:51 smithi135 ceph-mon[50158]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T01:13:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:51 smithi135 ceph-mon[50158]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T01:13:52.664 INFO:teuthology.orchestra.run.smithi104.stdout: 5 200915 11.13 MB/sec execute 314 sec latency 2173.075 ms 2024-04-27T01:13:52.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:52 smithi104 ceph-mon[63925]: pgmap v76: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 360 op/s; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:52.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:52 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 227/1668 objects degraded (13.609%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:53.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:52 smithi135 ceph-mon[50158]: pgmap v76: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 360 op/s; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:53.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:52 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 227/1668 objects degraded (13.609%), 29 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 201434 11.15 MB/sec execute 315 sec latency 26.379 ms 2024-04-27T01:13:53.838 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:53 smithi104 ceph-mon[63925]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T01:13:54.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:53 smithi135 ceph-mon[50158]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T01:13:54.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 202046 11.18 MB/sec execute 316 sec latency 549.240 ms 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: pgmap v79: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:54.936 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:54.937 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:54 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: pgmap v79: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 227/1668 objects degraded (13.609%) 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:55.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:55.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:13:55.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:55.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:55.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:13:55.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:54 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:13:55.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 202409 11.18 MB/sec execute 317 sec latency 711.812 ms 2024-04-27T01:13:55.953 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: pgmap v81: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 606 op/s; 214/1605 objects degraded (13.333%); 2.3 MiB/s, 24 objects/s recovering 2024-04-27T01:13:55.954 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: pgmap v82: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.4 MiB/s wr, 674 op/s; 214/1605 objects degraded (13.333%); 2.6 MiB/s, 27 objects/s recovering 2024-04-27T01:13:55.954 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:13:55.954 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-27T01:13:55.954 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: osdmap e85: 8 total, 8 up, 8 in 2024-04-27T01:13:55.954 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:55 smithi104 ceph-mon[63925]: osdmap e86: 8 total, 8 up, 8 in 2024-04-27T01:13:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: pgmap v81: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 606 op/s; 214/1605 objects degraded (13.333%); 2.3 MiB/s, 24 objects/s recovering 2024-04-27T01:13:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: pgmap v82: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.4 MiB/s wr, 674 op/s; 214/1605 objects degraded (13.333%); 2.6 MiB/s, 27 objects/s recovering 2024-04-27T01:13:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:13:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-27T01:13:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: osdmap e85: 8 total, 8 up, 8 in 2024-04-27T01:13:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:55 smithi135 ceph-mon[50158]: osdmap e86: 8 total, 8 up, 8 in 2024-04-27T01:13:56.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 202760 11.17 MB/sec execute 318 sec latency 54.042 ms 2024-04-27T01:13:57.194 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:13:57.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 203055 11.16 MB/sec execute 319 sec latency 652.654 ms 2024-04-27T01:13:57.993 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:57 smithi104 ceph-mon[63925]: pgmap v85: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.4 MiB/s wr, 674 op/s; 214/1605 objects degraded (13.333%); 2.6 MiB/s, 27 objects/s recovering 2024-04-27T01:13:57.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:57 smithi104 ceph-mon[63925]: osdmap e87: 8 total, 8 up, 8 in 2024-04-27T01:13:57.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:57 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 214/1605 objects degraded (13.333%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:57.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:57.994 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:58.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:57 smithi135 ceph-mon[50158]: pgmap v85: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.4 MiB/s wr, 674 op/s; 214/1605 objects degraded (13.333%); 2.6 MiB/s, 27 objects/s recovering 2024-04-27T01:13:58.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:57 smithi135 ceph-mon[50158]: osdmap e87: 8 total, 8 up, 8 in 2024-04-27T01:13:58.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:57 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 214/1605 objects degraded (13.333%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:13:58.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:13:58.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (4m) 10s ago 13m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (60s) 10s ago 13m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (53s) 48s ago 13m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 10s ago 13m - - 2024-04-27T01:13:58.376 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (7m) 10s ago 7m 40.4M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (7m) 48s ago 7m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (3m) 10s ago 14m 502M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (2m) 48s ago 13m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (2m) 10s ago 15m 48.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (118s) 48s ago 12m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (7m) 10s ago 7m 265M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (7m) 48s ago 7m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (4m) 10s ago 13m 18.9M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (3m) 48s ago 12m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (19s) 10s ago 12m 12.7M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (11m) 10s ago 11m 331M 1936M 17.2.0 e1d6a67b021e 4ce0c610c277 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (11m) 10s ago 11m 532M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (10m) 10s ago 10m 602M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (10m) 48s ago 10m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:13:58.377 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (10m) 48s ago 10m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:13:58.378 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (9m) 48s ago 9m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:13:58.378 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (9m) 48s ago 9m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:13:58.378 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (2m) 10s ago 12m 66.2M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:13:58.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 203386 11.14 MB/sec execute 320 sec latency 853.430 ms 2024-04-27T01:13:58.997 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:13:58.997 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:13:58.997 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:13:58.997 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-27T01:13:58.998 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T01:13:58.999 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:13:58.999 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:13:59.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:58 smithi104 ceph-mon[63925]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:59.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:58 smithi104 ceph-mon[63925]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:59.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:58 smithi104 ceph-mon[63925]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T01:13:59.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:58 smithi135 ceph-mon[50158]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:59.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:58 smithi135 ceph-mon[50158]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:13:59.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:58 smithi135 ceph-mon[50158]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "7/23 daemons upgraded", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:13:59.539 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:13:59.540 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:13:59.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 203760 11.13 MB/sec execute 321 sec latency 344.702 ms 2024-04-27T01:14:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:59 smithi104 ceph-mon[63925]: pgmap v88: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 214/1605 objects degraded (13.333%) 2024-04-27T01:14:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:59 smithi104 ceph-mon[63925]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:00.032 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:59 smithi104 ceph-mon[63925]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T01:14:00.033 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:13:59 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1122407979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:00.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:59 smithi135 ceph-mon[50158]: pgmap v88: 97 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 214/1605 objects degraded (13.333%) 2024-04-27T01:14:00.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:59 smithi135 ceph-mon[50158]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:00.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:59 smithi135 ceph-mon[50158]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T01:14:00.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:13:59 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1122407979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 214/1605 objects degraded (13.333%), 9 pgs degraded 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 214/1605 objects degraded (13.333%), 9 pgs degraded 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1 is active+recovering+undersized+degraded+remapped, acting [3,4] 2024-04-27T01:14:00.135 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.6 is active+recovery_wait+undersized+degraded+remapped, acting [1,4] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.7 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.b is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.10 is active+recovery_wait+undersized+degraded+remapped, acting [6,5] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.12 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.17 is active+recovery_wait+undersized+degraded+remapped, acting [3,5] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.18 is active+recovery_wait+undersized+degraded+remapped, acting [3,1] 2024-04-27T01:14:00.136 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1b is active+recovery_wait+undersized+degraded+remapped, acting [4,7] 2024-04-27T01:14:00.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 204230 11.13 MB/sec execute 322 sec latency 342.517 ms 2024-04-27T01:14:01.040 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:00 smithi104 ceph-mon[63925]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:01.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:00 smithi104 ceph-mon[63925]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T01:14:01.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:00 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/4062155282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:14:01.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:00 smithi135 ceph-mon[50158]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:01.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:00 smithi135 ceph-mon[50158]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T01:14:01.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:00 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/4062155282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:14:01.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 204704 11.14 MB/sec execute 323 sec latency 536.306 ms 2024-04-27T01:14:02.054 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:01 smithi104 ceph-mon[63925]: pgmap v91: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 754 op/s; 12/1707 objects degraded (0.703%); 1024 KiB/s, 48 objects/s recovering 2024-04-27T01:14:02.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:01 smithi104 ceph-mon[63925]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T01:14:02.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:01 smithi135 ceph-mon[50158]: pgmap v91: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 754 op/s; 12/1707 objects degraded (0.703%); 1024 KiB/s, 48 objects/s recovering 2024-04-27T01:14:02.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:01 smithi135 ceph-mon[50158]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T01:14:02.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 205154 11.15 MB/sec execute 324 sec latency 36.687 ms 2024-04-27T01:14:03.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:02 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 12/1707 objects degraded (0.703%), 1 pg degraded (PG_DEGRADED) 2024-04-27T01:14:03.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:02 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 12/1707 objects degraded (0.703%), 1 pg degraded (PG_DEGRADED) 2024-04-27T01:14:03.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 205505 11.15 MB/sec execute 325 sec latency 34.292 ms 2024-04-27T01:14:04.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:03 smithi104 ceph-mon[63925]: pgmap v93: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.2 MiB/s wr, 684 op/s; 12/1707 objects degraded (0.703%); 928 KiB/s, 44 objects/s recovering 2024-04-27T01:14:04.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:03 smithi135 ceph-mon[50158]: pgmap v93: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.2 MiB/s wr, 684 op/s; 12/1707 objects degraded (0.703%); 928 KiB/s, 44 objects/s recovering 2024-04-27T01:14:04.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 205864 11.16 MB/sec execute 326 sec latency 44.718 ms 2024-04-27T01:14:05.110 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:04 smithi104 ceph-mon[63925]: pgmap v94: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 999 op/s; 683 KiB/s, 35 objects/s recovering 2024-04-27T01:14:05.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:04 smithi135 ceph-mon[50158]: pgmap v94: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 999 op/s; 683 KiB/s, 35 objects/s recovering 2024-04-27T01:14:05.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 206172 11.15 MB/sec execute 327 sec latency 42.474 ms 2024-04-27T01:14:06.136 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:05 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1707 objects degraded (0.703%), 1 pg degraded) 2024-04-27T01:14:06.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:05 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1707 objects degraded (0.703%), 1 pg degraded) 2024-04-27T01:14:06.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 206411 11.13 MB/sec execute 328 sec latency 54.189 ms 2024-04-27T01:14:06.901 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:06 smithi104 ceph-mon[63925]: pgmap v95: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 402 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:14:07.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:06 smithi135 ceph-mon[50158]: pgmap v95: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 402 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:14:07.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 206622 11.11 MB/sec execute 329 sec latency 104.128 ms 2024-04-27T01:14:08.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 206850 11.08 MB/sec execute 330 sec latency 49.366 ms 2024-04-27T01:14:09.507 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:09 smithi104 ceph-mon[63925]: pgmap v96: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.6 MiB/s wr, 354 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:14:09.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:09 smithi135 ceph-mon[50158]: pgmap v96: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.6 MiB/s wr, 354 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:14:09.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 207097 11.07 MB/sec execute 331 sec latency 97.146 ms 2024-04-27T01:14:10.541 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:14:10.541 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:10 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:14:10.541 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:10 smithi104 ceph-mon[63925]: Upgrade: osd.1 is safe to restart 2024-04-27T01:14:10.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:14:10.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:10 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-27T01:14:10.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:10 smithi135 ceph-mon[50158]: Upgrade: osd.1 is safe to restart 2024-04-27T01:14:10.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 207382 11.06 MB/sec execute 332 sec latency 44.560 ms 2024-04-27T01:14:11.315 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:11 smithi104 ceph-mon[63925]: pgmap v97: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 519 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:11.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:11 smithi135 ceph-mon[50158]: pgmap v97: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 519 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 207702 11.05 MB/sec execute 333 sec latency 55.668 ms 2024-04-27T01:14:12.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 208026 11.03 MB/sec execute 334 sec latency 48.726 ms 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: Upgrade: Updating osd.1 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: Deploying daemon osd.1 on smithi104 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: pgmap v98: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 456 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:13.350 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: Upgrade: Updating osd.1 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: Deploying daemon osd.1 on smithi104 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: pgmap v98: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 456 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:13.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:13.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 208314 11.04 MB/sec execute 335 sec latency 58.686 ms 2024-04-27T01:14:14.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 208648 11.03 MB/sec execute 336 sec latency 43.828 ms 2024-04-27T01:14:15.388 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:15 smithi104 ceph-mon[63925]: pgmap v99: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.9 MiB/s wr, 612 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:15.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:15 smithi135 ceph-mon[50158]: pgmap v99: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.9 MiB/s wr, 612 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:14:15.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 209041 11.04 MB/sec execute 337 sec latency 47.904 ms 2024-04-27T01:14:16.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 209401 11.05 MB/sec execute 338 sec latency 59.342 ms 2024-04-27T01:14:17.451 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:17 smithi104 ceph-mon[63925]: pgmap v100: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 364 op/s 2024-04-27T01:14:17.452 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:17 smithi104 systemd[1]: Stopping Ceph osd.1 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:14:17.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:17 smithi135 ceph-mon[50158]: pgmap v100: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 364 op/s 2024-04-27T01:14:17.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 209662 11.03 MB/sec execute 339 sec latency 45.153 ms 2024-04-27T01:14:17.711 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:17 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[35086]: 2024-04-27T01:14:17.642+0000 7f2b3ec8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:14:17.712 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:17 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[35086]: 2024-04-27T01:14:17.642+0000 7f2b3ec8e700 -1 osd.1 91 *** Got signal Terminated *** 2024-04-27T01:14:17.712 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:17 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[35086]: 2024-04-27T01:14:17.642+0000 7f2b3ec8e700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:14:18.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:18 smithi135 ceph-mon[50158]: osd.1 marked itself down and dead 2024-04-27T01:14:18.648 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:18 smithi104 ceph-mon[63925]: osd.1 marked itself down and dead 2024-04-27T01:14:18.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 209836 11.01 MB/sec execute 340 sec latency 355.427 ms 2024-04-27T01:14:19.236 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:18 smithi104 podman[72084]: 2024-04-27 01:14:18.974624715 +0000 UTC m=+1.552573388 container died 4ce0c610c27737b019be5dbbc809903920cf67db76cf414b0e78171465c3f3e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-27T01:14:19.486 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:19 smithi104 ceph-mon[63925]: pgmap v101: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 364 op/s 2024-04-27T01:14:19.486 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:19 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:14:19.487 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:19 smithi104 ceph-mon[63925]: osdmap e92: 8 total, 7 up, 8 in 2024-04-27T01:14:19.487 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:19 smithi104 podman[72084]: 2024-04-27 01:14:19.363966053 +0000 UTC m=+1.941914725 container cleanup 4ce0c610c27737b019be5dbbc809903920cf67db76cf414b0e78171465c3f3e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, architecture=x86_64, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-27T01:14:19.487 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:19 smithi104 bash[72084]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1 2024-04-27T01:14:19.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:19 smithi135 ceph-mon[50158]: pgmap v101: 97 pgs: 97 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.4 MiB/s wr, 364 op/s 2024-04-27T01:14:19.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:19 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:14:19.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:19 smithi135 ceph-mon[50158]: osdmap e92: 8 total, 7 up, 8 in 2024-04-27T01:14:19.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 209930 10.98 MB/sec execute 341 sec latency 1136.429 ms 2024-04-27T01:14:19.737 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:19 smithi104 podman[72103]: 2024-04-27 01:14:19.582366921 +0000 UTC m=+0.607710394 container remove 4ce0c610c27737b019be5dbbc809903920cf67db76cf414b0e78171465c3f3e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, version=8, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-27T01:14:20.241 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.019892752 +0000 UTC m=+0.026216764 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:14:20.242 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.183523673 +0000 UTC m=+0.189847714 container create 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-27T01:14:20.492 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:20 smithi104 ceph-mon[63925]: osdmap e93: 8 total, 7 up, 8 in 2024-04-27T01:14:20.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:20 smithi135 ceph-mon[50158]: osdmap e93: 8 total, 7 up, 8 in 2024-04-27T01:14:20.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 210280 10.97 MB/sec execute 342 sec latency 39.606 ms 2024-04-27T01:14:20.748 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.542348693 +0000 UTC m=+0.548672696 container init 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-27T01:14:20.749 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.552814087 +0000 UTC m=+0.559138081 container start 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:14:20.749 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.7439615 +0000 UTC m=+0.750285504 container attach 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T01:14:21.000 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:20 smithi104 podman[72193]: 2024-04-27 01:14:20.979314467 +0000 UTC m=+0.985638463 container died 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-27T01:14:21.474 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:21 smithi104 ceph-mon[63925]: pgmap v104: 97 pgs: 30 peering, 67 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 555 op/s 2024-04-27T01:14:21.474 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:21 smithi104 ceph-mon[63925]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:14:21.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:21 smithi135 ceph-mon[50158]: pgmap v104: 97 pgs: 30 peering, 67 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 555 op/s 2024-04-27T01:14:21.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:21 smithi135 ceph-mon[50158]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:14:21.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 210641 10.96 MB/sec execute 343 sec latency 40.785 ms 2024-04-27T01:14:22.268 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 podman[72218]: 2024-04-27 01:14:22.133791733 +0000 UTC m=+1.145075024 container remove 1094bbf1a41f30bc6e087b4dc7028bdea4628004aa581885f67a5fdbb491853c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-27T01:14:22.268 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-27T01:14:22.268 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 systemd[1]: Stopped Ceph osd.1 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:14:22.268 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.1.service: Consumed 3min 48.915s CPU time. 2024-04-27T01:14:22.542 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 systemd[1]: Starting Ceph osd.1 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:14:22.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 210991 10.95 MB/sec execute 344 sec latency 50.820 ms 2024-04-27T01:14:22.799 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 podman[72316]: 2024-04-27 01:14:22.655393291 +0000 UTC m=+0.024136913 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:14:22.799 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:22 smithi104 podman[72316]: 2024-04-27 01:14:22.798729805 +0000 UTC m=+0.167473429 container create d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:14:23.554 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:23 smithi104 ceph-mon[63925]: pgmap v105: 97 pgs: 30 peering, 67 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 286 op/s 2024-04-27T01:14:23.554 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:23 smithi104 podman[72316]: 2024-04-27 01:14:23.461029416 +0000 UTC m=+0.829773046 container init d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:14:23.555 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:23 smithi104 podman[72316]: 2024-04-27 01:14:23.471233059 +0000 UTC m=+0.839976686 container start d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD) 2024-04-27T01:14:23.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:23 smithi135 ceph-mon[50158]: pgmap v105: 97 pgs: 30 peering, 67 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 286 op/s 2024-04-27T01:14:23.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 211366 10.95 MB/sec execute 345 sec latency 53.450 ms 2024-04-27T01:14:23.823 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:23 smithi104 podman[72316]: 2024-04-27 01:14:23.627511389 +0000 UTC m=+0.996255014 container attach d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-27T01:14:24.088 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 bash[72316]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-27T01:14:24.089 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T01:14:24.090 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate[72331]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-27T01:14:24.351 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:24 smithi104 podman[72316]: 2024-04-27 01:14:24.108362825 +0000 UTC m=+1.477106450 container died d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:14:24.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 185/1647 objects degraded (11.233%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:24.603 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:24 smithi104 ceph-mon[63925]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-27T01:14:24.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 211784 10.95 MB/sec execute 346 sec latency 52.090 ms 2024-04-27T01:14:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:24 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 185/1647 objects degraded (11.233%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:24 smithi135 ceph-mon[50158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-27T01:14:25.620 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:25 smithi104 ceph-mon[63925]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 565 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:25.621 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:25 smithi104 podman[72457]: 2024-04-27 01:14:25.571019832 +0000 UTC m=+1.447963723 container remove d6c2eda79989c1bae9d76d673cd49cb8b3ac59c97218092b463fa06dee151803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T01:14:25.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 212193 10.96 MB/sec execute 347 sec latency 48.064 ms 2024-04-27T01:14:25.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:25 smithi135 ceph-mon[50158]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 565 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:25.886 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:25 smithi104 podman[72515]: 2024-04-27 01:14:25.732927336 +0000 UTC m=+0.024132496 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:14:25.887 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:25 smithi104 podman[72515]: 2024-04-27 01:14:25.846540999 +0000 UTC m=+0.137746438 container create da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:14:26.397 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 podman[72515]: 2024-04-27 01:14:26.177471803 +0000 UTC m=+0.468676958 container init da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-27T01:14:26.397 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 podman[72515]: 2024-04-27 01:14:26.184205975 +0000 UTC m=+0.475411122 container start da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-27T01:14:26.397 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 ceph-osd[72532]: -- 172.21.15.104:0/3537196005 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb8334f680 con 0x55fb83318c00 2024-04-27T01:14:26.661 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 bash[72515]: da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e 2024-04-27T01:14:26.661 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 systemd[1]: Started Ceph osd.1 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:14:26.665 INFO:teuthology.orchestra.run.smithi104.stdout: 5 212559 10.96 MB/sec execute 348 sec latency 37.357 ms 2024-04-27T01:14:26.913 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:26 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:14:26.760+0000 7f1c3fecf7c0 -1 Falling back to public interface 2024-04-27T01:14:27.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 212965 10.98 MB/sec execute 349 sec latency 38.076 ms 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: pgmap v107: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 565 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.674 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: pgmap v107: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 565 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:27.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:27.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:28.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 213345 10.97 MB/sec execute 350 sec latency 35.492 ms 2024-04-27T01:14:29.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 213760 10.96 MB/sec execute 351 sec latency 61.863 ms 2024-04-27T01:14:29.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:29 smithi104 ceph-mon[63925]: pgmap v108: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 455 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:29.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:29 smithi135 ceph-mon[50158]: pgmap v108: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 328 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 455 op/s; 185/1647 objects degraded (11.233%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:30.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 214140 10.95 MB/sec execute 352 sec latency 93.076 ms 2024-04-27T01:14:30.777 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:14:30.807 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:30 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 197/1731 objects degraded (11.381%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:30.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:30 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 197/1731 objects degraded (11.381%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:31.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 214442 10.94 MB/sec execute 353 sec latency 50.270 ms 2024-04-27T01:14:31.821 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:31 smithi104 ceph-mon[63925]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 472 op/s; 197/1731 objects degraded (11.381%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:31.821 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:31 smithi104 ceph-mon[63925]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:31 smithi135 ceph-mon[50158]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 472 op/s; 197/1731 objects degraded (11.381%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:31 smithi135 ceph-mon[50158]: from='client.44187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (5m) 44s ago 14m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (93s) 44s ago 14m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (87s) 81s ago 13m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 44s ago 14m - - 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (8m) 44s ago 8m 40.4M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (8m) 81s ago 8m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (4m) 44s ago 15m 502M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:14:31.873 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (3m) 81s ago 13m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (3m) 44s ago 15m 48.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (2m) 81s ago 13m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (8m) 44s ago 8m 265M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (8m) 81s ago 8m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (4m) 44s ago 13m 18.9M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (4m) 81s ago 13m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (52s) 44s ago 12m 12.7M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 starting - - - 1936M 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (11m) 44s ago 11m 532M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (11m) 44s ago 11m 602M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (11m) 81s ago 11m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (10m) 81s ago 10m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (10m) 81s ago 10m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:14:31.874 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (9m) 81s ago 9m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:14:31.875 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (3m) 44s ago 13m 66.2M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:14:32.539 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:14:32.539 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:14:32.539 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:14:32.539 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:14:32.539 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 1 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:14:32.540 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:14:32.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 214792 10.93 MB/sec execute 354 sec latency 58.661 ms 2024-04-27T01:14:32.846 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:32 smithi104 ceph-mon[63925]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:32.846 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:32 smithi104 ceph-mon[63925]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:32.846 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:32 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3755003368' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:32.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:32 smithi135 ceph-mon[50158]: from='client.34246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:32.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:32 smithi135 ceph-mon[50158]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:32.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:32 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3755003368' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "7/23 daemons upgraded", 2024-04-27T01:14:33.093 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:14:33.094 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:14:33.094 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:14:33.632 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:33 smithi104 ceph-mon[63925]: pgmap v110: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 430 op/s; 197/1731 objects degraded (11.381%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:33.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 215121 10.92 MB/sec execute 355 sec latency 52.931 ms 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 197/1731 objects degraded (11.381%), 23 pgs degraded 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: osd.1 (root=default,host=smithi104) is down 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 197/1731 objects degraded (11.381%), 23 pgs degraded 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-04-27T01:14:33.758 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-04-27T01:14:33.759 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-04-27T01:14:33.760 INFO:teuthology.orchestra.run.smithi104.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-04-27T01:14:33.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:33 smithi135 ceph-mon[50158]: pgmap v110: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 430 op/s; 197/1731 objects degraded (11.381%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:34.636 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:34 smithi104 ceph-mon[63925]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:34.636 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:34 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3728763375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:14:34.636 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:34 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:14:34.598+0000 7f1c3fecf7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-27T01:14:34.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 215503 10.93 MB/sec execute 356 sec latency 55.338 ms 2024-04-27T01:14:35.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:34 smithi135 ceph-mon[50158]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:14:35.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:34 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3728763375' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:14:35.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:35 smithi104 ceph-mon[63925]: pgmap v111: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 645 op/s; 185/1641 objects degraded (11.274%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:35.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 215924 10.93 MB/sec execute 357 sec latency 41.340 ms 2024-04-27T01:14:35.913 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:35 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:14:35.896+0000 7f1c3fecf7c0 -1 osd.1 91 log_to_monitors true 2024-04-27T01:14:36.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:35 smithi135 ceph-mon[50158]: pgmap v111: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.0 MiB/s wr, 645 op/s; 185/1641 objects degraded (11.274%); 0 B/s, 0 objects/s recovering 2024-04-27T01:14:36.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 216283 10.94 MB/sec execute 358 sec latency 47.880 ms 2024-04-27T01:14:36.682 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:36 smithi104 ceph-mon[63925]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:14:36.682 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:36.683 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:36 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:37.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:36 smithi135 ceph-mon[50158]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:14:37.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:37.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:36 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:37.228 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:14:37.135+0000 7f1c2d31f700 -1 osd.1 91 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:14:37.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 216533 10.94 MB/sec execute 359 sec latency 122.727 ms 2024-04-27T01:14:37.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: pgmap v112: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s; 185/1641 objects degraded (11.274%) 2024-04-27T01:14:37.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:14:37.747 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: osdmap e94: 8 total, 7 up, 8 in 2024-04-27T01:14:37.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:14:37.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' 2024-04-27T01:14:37.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 185/1641 objects degraded (11.274%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:37.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:37.748 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:38.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: pgmap v112: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s; 185/1641 objects degraded (11.274%) 2024-04-27T01:14:38.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: osdmap e94: 8 total, 7 up, 8 in 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: from='osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866]' entity='osd.1' 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 185/1641 objects degraded (11.274%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:38.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:38.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 216908 10.92 MB/sec execute 360 sec latency 513.147 ms 2024-04-27T01:14:38.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:38 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:14:38.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:38 smithi104 ceph-mon[63925]: osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866] boot 2024-04-27T01:14:38.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:38 smithi104 ceph-mon[63925]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T01:14:38.759 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:14:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:38 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:14:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:38 smithi135 ceph-mon[50158]: osd.1 [v2:172.21.15.104:6810/3679392866,v1:172.21.15.104:6811/3679392866] boot 2024-04-27T01:14:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:38 smithi135 ceph-mon[50158]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T01:14:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:14:39.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 216923 10.89 MB/sec execute 361 sec latency 1513.176 ms 2024-04-27T01:14:39.792 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:39 smithi104 ceph-mon[63925]: pgmap v115: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.6 MiB/s wr, 323 op/s; 185/1641 objects degraded (11.274%) 2024-04-27T01:14:39.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:39 smithi104 ceph-mon[63925]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T01:14:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:39 smithi135 ceph-mon[50158]: pgmap v115: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 329 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.6 MiB/s wr, 323 op/s; 185/1641 objects degraded (11.274%) 2024-04-27T01:14:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:39 smithi135 ceph-mon[50158]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T01:14:40.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 217029 10.87 MB/sec execute 362 sec latency 2144.892 ms 2024-04-27T01:14:41.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:41 smithi104 ceph-mon[63925]: osdmap e97: 8 total, 8 up, 8 in 2024-04-27T01:14:41.308 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:41 smithi104 ceph-mon[63925]: pgmap v118: 97 pgs: 26 peering, 9 active+undersized+degraded, 4 active+undersized, 58 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 603 op/s; 69/1707 objects degraded (4.042%) 2024-04-27T01:14:41.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:41 smithi135 ceph-mon[50158]: osdmap e97: 8 total, 8 up, 8 in 2024-04-27T01:14:41.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:41 smithi135 ceph-mon[50158]: pgmap v118: 97 pgs: 26 peering, 9 active+undersized+degraded, 4 active+undersized, 58 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 603 op/s; 69/1707 objects degraded (4.042%) 2024-04-27T01:14:41.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 217459 10.87 MB/sec execute 363 sec latency 42.078 ms 2024-04-27T01:14:42.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:42 smithi104 ceph-mon[63925]: osdmap e98: 8 total, 8 up, 8 in 2024-04-27T01:14:42.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:42 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 69/1707 objects degraded (4.042%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:42.341 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:42 smithi104 ceph-mon[63925]: osdmap e99: 8 total, 8 up, 8 in 2024-04-27T01:14:42.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:42 smithi135 ceph-mon[50158]: osdmap e98: 8 total, 8 up, 8 in 2024-04-27T01:14:42.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:42 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 69/1707 objects degraded (4.042%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:42.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:42 smithi135 ceph-mon[50158]: osdmap e99: 8 total, 8 up, 8 in 2024-04-27T01:14:42.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 217848 10.86 MB/sec execute 364 sec latency 45.633 ms 2024-04-27T01:14:43.625 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: pgmap v121: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 606 op/s; 44/1707 objects degraded (2.578%); 22 B/s, 72 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: pgmap v122: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s wr, 3 op/s; 44/1707 objects degraded (2.578%); 27 B/s, 87 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: pgmap v123: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s wr, 5 op/s; 44/1707 objects degraded (2.578%); 38 B/s, 124 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.626 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-27T01:14:43.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:43 smithi104 ceph-mon[63925]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T01:14:43.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 218289 10.85 MB/sec execute 365 sec latency 221.019 ms 2024-04-27T01:14:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: pgmap v121: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 606 op/s; 44/1707 objects degraded (2.578%); 22 B/s, 72 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: pgmap v122: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s wr, 3 op/s; 44/1707 objects degraded (2.578%); 27 B/s, 87 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: pgmap v123: 97 pgs: 27 peering, 3 active+undersized+degraded, 67 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s wr, 5 op/s; 44/1707 objects degraded (2.578%); 38 B/s, 124 keys/s, 0 objects/s recovering 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:43.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:43.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-27T01:14:43.865 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:43 smithi135 ceph-mon[50158]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T01:14:44.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 218604 10.85 MB/sec execute 366 sec latency 50.958 ms 2024-04-27T01:14:45.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:45 smithi104 ceph-mon[63925]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T01:14:45.166 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:45 smithi104 ceph-mon[63925]: pgmap v126: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 7.9 MiB/s wr, 933 op/s; 133/1686 objects degraded (7.888%); 1.5 MiB/s, 64 objects/s recovering 2024-04-27T01:14:45.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:45 smithi135 ceph-mon[50158]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T01:14:45.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:45 smithi135 ceph-mon[50158]: pgmap v126: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 7.9 MiB/s wr, 933 op/s; 133/1686 objects degraded (7.888%); 1.5 MiB/s, 64 objects/s recovering 2024-04-27T01:14:45.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 218952 10.86 MB/sec execute 367 sec latency 551.092 ms 2024-04-27T01:14:46.262 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:46 smithi104 ceph-mon[63925]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T01:14:46.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:46 smithi135 ceph-mon[50158]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T01:14:46.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 219316 10.85 MB/sec execute 368 sec latency 962.009 ms 2024-04-27T01:14:47.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:47 smithi104 ceph-mon[63925]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T01:14:47.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:47 smithi104 ceph-mon[63925]: pgmap v129: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 541 op/s; 133/1686 objects degraded (7.888%); 873 KiB/s, 37 objects/s recovering 2024-04-27T01:14:47.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:47 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 133/1686 objects degraded (7.888%), 6 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:47.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:47 smithi104 ceph-mon[63925]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T01:14:47.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:47 smithi135 ceph-mon[50158]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T01:14:47.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:47 smithi135 ceph-mon[50158]: pgmap v129: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 541 op/s; 133/1686 objects degraded (7.888%); 873 KiB/s, 37 objects/s recovering 2024-04-27T01:14:47.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:47 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 133/1686 objects degraded (7.888%), 6 pgs degraded (PG_DEGRADED) 2024-04-27T01:14:47.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:47 smithi135 ceph-mon[50158]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T01:14:47.667 INFO:teuthology.orchestra.run.smithi104.stdout: 5 219610 10.85 MB/sec execute 369 sec latency 433.980 ms 2024-04-27T01:14:48.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 219858 10.84 MB/sec execute 370 sec latency 480.138 ms 2024-04-27T01:14:49.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:49 smithi104 ceph-mon[63925]: osdmap e105: 8 total, 8 up, 8 in 2024-04-27T01:14:49.326 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:49 smithi104 ceph-mon[63925]: pgmap v132: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.9 MiB/s wr, 488 op/s; 78/1662 objects degraded (4.693%); 1024 KiB/s, 17 objects/s recovering 2024-04-27T01:14:49.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:49 smithi135 ceph-mon[50158]: osdmap e105: 8 total, 8 up, 8 in 2024-04-27T01:14:49.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:49 smithi135 ceph-mon[50158]: pgmap v132: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.9 MiB/s wr, 488 op/s; 78/1662 objects degraded (4.693%); 1024 KiB/s, 17 objects/s recovering 2024-04-27T01:14:49.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 220183 10.84 MB/sec execute 371 sec latency 32.908 ms 2024-04-27T01:14:50.352 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:50 smithi104 ceph-mon[63925]: osdmap e106: 8 total, 8 up, 8 in 2024-04-27T01:14:50.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:50 smithi135 ceph-mon[50158]: osdmap e106: 8 total, 8 up, 8 in 2024-04-27T01:14:50.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 220554 10.84 MB/sec execute 372 sec latency 38.253 ms 2024-04-27T01:14:51.380 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:51 smithi104 ceph-mon[63925]: pgmap v134: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.6 MiB/s wr, 532 op/s; 27/1674 objects degraded (1.613%); 957 KiB/s, 28 objects/s recovering 2024-04-27T01:14:51.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:51 smithi135 ceph-mon[50158]: pgmap v134: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.6 MiB/s wr, 532 op/s; 27/1674 objects degraded (1.613%); 957 KiB/s, 28 objects/s recovering 2024-04-27T01:14:51.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 220833 10.83 MB/sec execute 373 sec latency 55.365 ms 2024-04-27T01:14:52.157 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:52 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 27/1674 objects degraded (1.613%), 1 pg degraded (PG_DEGRADED) 2024-04-27T01:14:52.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:52 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 27/1674 objects degraded (1.613%), 1 pg degraded (PG_DEGRADED) 2024-04-27T01:14:52.666 INFO:teuthology.orchestra.run.smithi104.stdout: 5 221091 10.81 MB/sec execute 374 sec latency 71.806 ms 2024-04-27T01:14:53.191 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:53 smithi104 ceph-mon[63925]: pgmap v135: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.6 MiB/s wr, 399 op/s; 682 KiB/s, 24 objects/s recovering 2024-04-27T01:14:53.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:53 smithi135 ceph-mon[50158]: pgmap v135: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.6 MiB/s wr, 399 op/s; 682 KiB/s, 24 objects/s recovering 2024-04-27T01:14:53.667 INFO:teuthology.orchestra.run.smithi104.stdout: 5 221325 10.80 MB/sec execute 375 sec latency 53.814 ms 2024-04-27T01:14:54.216 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:54 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1674 objects degraded (1.613%), 1 pg degraded) 2024-04-27T01:14:54.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:54 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1674 objects degraded (1.613%), 1 pg degraded) 2024-04-27T01:14:54.668 INFO:teuthology.orchestra.run.smithi104.stdout: 5 221570 10.77 MB/sec execute 376 sec latency 78.840 ms 2024-04-27T01:14:55.263 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:55 smithi104 ceph-mon[63925]: pgmap v136: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 622 op/s; 0 B/s, 17 objects/s recovering 2024-04-27T01:14:55.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:55 smithi135 ceph-mon[50158]: pgmap v136: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 622 op/s; 0 B/s, 17 objects/s recovering 2024-04-27T01:14:55.668 INFO:teuthology.orchestra.run.smithi104.stdout: 5 221889 10.76 MB/sec execute 377 sec latency 60.167 ms 2024-04-27T01:14:56.668 INFO:teuthology.orchestra.run.smithi104.stdout: 5 222189 10.76 MB/sec execute 378 sec latency 68.507 ms 2024-04-27T01:14:57.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:57 smithi104 ceph-mon[63925]: pgmap v137: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 547 op/s; 0 B/s, 15 objects/s recovering 2024-04-27T01:14:57.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:57.585 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:57.674 INFO:teuthology.orchestra.run.smithi104.stdout: 5 222737 10.78 MB/sec execute 379 sec latency 29.404 ms 2024-04-27T01:14:57.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:57 smithi135 ceph-mon[50158]: pgmap v137: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 547 op/s; 0 B/s, 15 objects/s recovering 2024-04-27T01:14:57.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:14:57.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:58.611 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:58.611 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:58 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:58.611 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:58 smithi104 ceph-mon[63925]: Upgrade: osd.2 is safe to restart 2024-04-27T01:14:58.674 INFO:teuthology.orchestra.run.smithi104.stdout: 5 223149 10.79 MB/sec execute 380 sec latency 39.407 ms 2024-04-27T01:14:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:58 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-27T01:14:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:58 smithi135 ceph-mon[50158]: Upgrade: osd.2 is safe to restart 2024-04-27T01:14:59.631 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:14:59 smithi104 ceph-mon[63925]: pgmap v138: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.1 MiB/s wr, 385 op/s; 0 B/s, 7 objects/s recovering 2024-04-27T01:14:59.674 INFO:teuthology.orchestra.run.smithi104.stdout: 5 223601 10.80 MB/sec execute 381 sec latency 47.347 ms 2024-04-27T01:14:59.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:14:59 smithi135 ceph-mon[50158]: pgmap v138: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.1 MiB/s wr, 385 op/s; 0 B/s, 7 objects/s recovering 2024-04-27T01:15:00.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 224002 10.79 MB/sec execute 382 sec latency 53.563 ms 2024-04-27T01:15:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: Upgrade: Updating osd.2 2024-04-27T01:15:01.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:01.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:15:01.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:01.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: Deploying daemon osd.2 on smithi104 2024-04-27T01:15:01.414 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:01 smithi104 ceph-mon[63925]: pgmap v139: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 368 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: Upgrade: Updating osd.2 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: Deploying daemon osd.2 on smithi104 2024-04-27T01:15:01.614 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:01 smithi135 ceph-mon[50158]: pgmap v139: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 368 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:15:01.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 224334 10.79 MB/sec execute 383 sec latency 53.609 ms 2024-04-27T01:15:02.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 224728 10.78 MB/sec execute 384 sec latency 40.293 ms 2024-04-27T01:15:03.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 225084 10.77 MB/sec execute 385 sec latency 47.756 ms 2024-04-27T01:15:03.979 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:03 smithi104 ceph-mon[63925]: pgmap v140: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 441 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:15:03.979 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:03 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:04.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:03 smithi135 ceph-mon[50158]: pgmap v140: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 441 op/s; 0 B/s, 2 objects/s recovering 2024-04-27T01:15:04.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:03 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:04.316 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:15:04.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 225417 10.76 MB/sec execute 386 sec latency 43.861 ms 2024-04-27T01:15:05.132 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:15:05.132 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (5m) 29s ago 14m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:15:05.132 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (2m) 29s ago 14m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:15:05.132 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (2m) 114s ago 14m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:15:05.132 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 29s ago 14m - - 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (9m) 29s ago 8m 40.6M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (8m) 114s ago 8m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (4m) 29s ago 16m 504M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (3m) 114s ago 14m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (3m) 29s ago 16m 50.0M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (3m) 114s ago 14m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (8m) 29s ago 8m 267M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (8m) 114s ago 8m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 29s ago 14m 22.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (5m) 114s ago 13m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (86s) 29s ago 13m 234M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (39s) 29s ago 12m 12.7M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (12m) 29s ago 12m 567M 1936M 17.2.0 e1d6a67b021e 3427434b0150 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (12m) 29s ago 11m 386M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:15:05.133 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (11m) 114s ago 11m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:15:05.134 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (11m) 114s ago 11m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:15:05.134 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (10m) 114s ago 10m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:15:05.134 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (10m) 114s ago 10m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:15:05.134 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (3m) 29s ago 13m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:05.615 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 6 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:15:05.616 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:15:05.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 225602 10.75 MB/sec execute 387 sec latency 54.378 ms 2024-04-27T01:15:05.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:05 smithi135 ceph-mon[50158]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:05.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:05 smithi135 ceph-mon[50158]: pgmap v141: 97 pgs: 97 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.9 MiB/s wr, 583 op/s 2024-04-27T01:15:05.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:05 smithi135 ceph-mon[50158]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:05.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:05 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/4210099565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:05.992 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "8/23 daemons upgraded", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:15:05.993 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:15:06.052 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:05 smithi104 ceph-mon[63925]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:06.052 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:05 smithi104 ceph-mon[63925]: pgmap v141: 97 pgs: 97 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.9 MiB/s wr, 583 op/s 2024-04-27T01:15:06.052 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:05 smithi104 ceph-mon[63925]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:06.052 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:05 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/4210099565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:06.525 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:15:06.526 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:15:06.526 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:15:06.563 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:06 smithi104 systemd[1]: Stopping Ceph osd.2 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:15:06.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 225811 10.74 MB/sec execute 388 sec latency 87.265 ms 2024-04-27T01:15:06.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:06 smithi104 ceph-mon[63925]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:06.819 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:06 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2757013076' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:15:07.091 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:06 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[38959]: 2024-04-27T01:15:06.865+0000 7f3b6d210700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:15:07.092 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:06 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[38959]: 2024-04-27T01:15:06.865+0000 7f3b6d210700 -1 osd.2 106 *** Got signal Terminated *** 2024-04-27T01:15:07.092 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:06 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[38959]: 2024-04-27T01:15:06.865+0000 7f3b6d210700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:15:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:06 smithi135 ceph-mon[50158]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:06 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2757013076' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:15:07.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 226153 10.74 MB/sec execute 389 sec latency 82.564 ms 2024-04-27T01:15:07.872 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:07 smithi104 ceph-mon[63925]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:07.872 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:07 smithi104 ceph-mon[63925]: pgmap v142: 97 pgs: 97 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 400 op/s 2024-04-27T01:15:07.872 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:07 smithi104 ceph-mon[63925]: osd.2 marked itself down and dead 2024-04-27T01:15:08.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:07 smithi135 ceph-mon[50158]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:08.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:07 smithi135 ceph-mon[50158]: pgmap v142: 97 pgs: 97 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 400 op/s 2024-04-27T01:15:08.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:07 smithi135 ceph-mon[50158]: osd.2 marked itself down and dead 2024-04-27T01:15:08.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 226274 10.72 MB/sec execute 390 sec latency 765.015 ms 2024-04-27T01:15:08.725 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:08 smithi104 podman[74773]: 2024-04-27 01:15:08.468259285 +0000 UTC m=+1.738033391 container died 3427434b01504f7aaed725ffa2d7f821a5d0fffe092554c9fba31b7f707f9728 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, name=centos-stream, ceph=True, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, version=8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=) 2024-04-27T01:15:08.985 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:08 smithi104 podman[74773]: 2024-04-27 01:15:08.945206621 +0000 UTC m=+2.214980732 container cleanup 3427434b01504f7aaed725ffa2d7f821a5d0fffe092554c9fba31b7f707f9728 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.buildah.version=1.19.8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, name=centos-stream) 2024-04-27T01:15:08.985 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:08 smithi104 bash[74773]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2 2024-04-27T01:15:08.986 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:08 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:15:08.986 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:08 smithi104 ceph-mon[63925]: osdmap e107: 8 total, 7 up, 8 in 2024-04-27T01:15:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:08 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:15:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:08 smithi135 ceph-mon[50158]: osdmap e107: 8 total, 7 up, 8 in 2024-04-27T01:15:09.488 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:09 smithi104 podman[74793]: 2024-04-27 01:15:09.418593185 +0000 UTC m=+0.941996695 container remove 3427434b01504f7aaed725ffa2d7f821a5d0fffe092554c9fba31b7f707f9728 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, build-date=2022-05-03T08:36:31.336870, release=754, architecture=x86_64, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, name=centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-27T01:15:09.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 226511 10.71 MB/sec execute 391 sec latency 941.452 ms 2024-04-27T01:15:10.003 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:09 smithi104 ceph-mon[63925]: pgmap v144: 97 pgs: 12 peering, 4 stale+active+clean, 81 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-27T01:15:10.003 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:09 smithi104 ceph-mon[63925]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:15:10.003 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:09 smithi104 ceph-mon[63925]: osdmap e108: 8 total, 7 up, 8 in 2024-04-27T01:15:10.003 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:09 smithi104 podman[74886]: 2024-04-27 01:15:09.708817139 +0000 UTC m=+0.021453853 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:15:10.003 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:09 smithi104 podman[74886]: 2024-04-27 01:15:09.850033825 +0000 UTC m=+0.162670507 container create f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-27T01:15:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:09 smithi135 ceph-mon[50158]: pgmap v144: 97 pgs: 12 peering, 4 stale+active+clean, 81 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 572 op/s 2024-04-27T01:15:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:09 smithi135 ceph-mon[50158]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:15:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:09 smithi135 ceph-mon[50158]: osdmap e108: 8 total, 7 up, 8 in 2024-04-27T01:15:10.195 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:10 smithi104 podman[74886]: 2024-04-27 01:15:10.195453551 +0000 UTC m=+0.508090234 container init f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T01:15:10.255 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:10 smithi104 podman[74886]: 2024-04-27 01:15:10.206286082 +0000 UTC m=+0.518922776 container start f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD) 2024-04-27T01:15:10.509 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:10 smithi104 podman[74886]: 2024-04-27 01:15:10.354684212 +0000 UTC m=+0.667320904 container attach f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-27T01:15:10.678 INFO:teuthology.orchestra.run.smithi104.stdout: 5 226777 10.71 MB/sec execute 392 sec latency 47.546 ms 2024-04-27T01:15:10.760 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:10 smithi104 podman[74886]: 2024-04-27 01:15:10.629314648 +0000 UTC m=+0.941951333 container died f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, ceph=True) 2024-04-27T01:15:11.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 227145 10.71 MB/sec execute 393 sec latency 36.952 ms 2024-04-27T01:15:12.044 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:11 smithi104 ceph-mon[63925]: pgmap v146: 97 pgs: 20 peering, 2 stale+active+clean, 75 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-27T01:15:12.044 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:11 smithi104 podman[74913]: 2024-04-27 01:15:11.813517031 +0000 UTC m=+1.177446201 container remove f4ffc23d7530aae9e26d6e61e77c0c98b4a96a0ea59f98cf483ad6878575f803 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T01:15:12.044 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:11 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-27T01:15:12.044 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:11 smithi104 systemd[1]: Stopped Ceph osd.2 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:15:12.044 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:11 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.2.service: Consumed 3min 13.758s CPU time. 2024-04-27T01:15:12.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:11 smithi135 ceph-mon[50158]: pgmap v146: 97 pgs: 20 peering, 2 stale+active+clean, 75 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-27T01:15:12.295 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 systemd[1]: Starting Ceph osd.2 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:15:12.546 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 podman[75010]: 2024-04-27 01:15:12.313244772 +0000 UTC m=+0.030633354 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:15:12.546 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 podman[75010]: 2024-04-27 01:15:12.465153758 +0000 UTC m=+0.182542322 container create 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-27T01:15:12.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 227435 10.70 MB/sec execute 394 sec latency 37.807 ms 2024-04-27T01:15:12.797 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 podman[75010]: 2024-04-27 01:15:12.76839358 +0000 UTC m=+0.485782155 container init 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:15:12.797 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 podman[75010]: 2024-04-27 01:15:12.774896198 +0000 UTC m=+0.492284761 container start 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-27T01:15:13.070 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:12 smithi104 podman[75010]: 2024-04-27 01:15:12.905857416 +0000 UTC m=+0.623245987 container attach 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:15:13.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-mon[63925]: pgmap v147: 97 pgs: 3 active+undersized+degraded, 20 peering, 2 active+undersized, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.7 MiB/s wr, 323 op/s; 31/1599 objects degraded (1.939%) 2024-04-27T01:15:13.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:13.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:13.588 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 31/1599 objects degraded (1.939%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate[75024]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T01:15:13.589 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-27T01:15:13.590 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T01:15:13.590 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 bash[75010]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-27T01:15:13.590 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 conmon[75024]: conmon 4d6a529d94b8ff7463ef : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d.scope/container/memory.events 2024-04-27T01:15:13.590 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:13 smithi104 podman[75010]: 2024-04-27 01:15:13.417995656 +0000 UTC m=+1.135384228 container died 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:15:13.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 227697 10.69 MB/sec execute 395 sec latency 58.520 ms 2024-04-27T01:15:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:13 smithi135 ceph-mon[50158]: pgmap v147: 97 pgs: 3 active+undersized+degraded, 20 peering, 2 active+undersized, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.7 MiB/s wr, 323 op/s; 31/1599 objects degraded (1.939%) 2024-04-27T01:15:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:13 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 31/1599 objects degraded (1.939%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:14.350 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 podman[75160]: 2024-04-27 01:15:14.259873084 +0000 UTC m=+0.838399647 container remove 4d6a529d94b8ff7463ef6154f91ccb52a399cc2454d127de7ff3b98948e9ab8d (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-27T01:15:14.602 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:14 smithi104 ceph-mon[63925]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-27T01:15:14.602 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 podman[75214]: 2024-04-27 01:15:14.416841146 +0000 UTC m=+0.025055729 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:15:14.602 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 podman[75214]: 2024-04-27 01:15:14.543395162 +0000 UTC m=+0.151609745 container create 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-27T01:15:14.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 227940 10.68 MB/sec execute 396 sec latency 58.356 ms 2024-04-27T01:15:14.853 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 podman[75214]: 2024-04-27 01:15:14.804630917 +0000 UTC m=+0.412845492 container init 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb) 2024-04-27T01:15:14.853 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 podman[75214]: 2024-04-27 01:15:14.811118514 +0000 UTC m=+0.419333097 container start 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-27T01:15:14.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:14 smithi135 ceph-mon[50158]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-27T01:15:15.108 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 bash[75214]: 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 2024-04-27T01:15:15.108 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:14 smithi104 systemd[1]: Started Ceph osd.2 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:15:15.613 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:15 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:15:15.393+0000 7fe9620947c0 -1 Falling back to public interface 2024-04-27T01:15:15.613 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:15 smithi104 ceph-mon[63925]: pgmap v148: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 156/1743 objects degraded (8.950%) 2024-04-27T01:15:15.613 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:15 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:15.613 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:15 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:15.613 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:15 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:15.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 228230 10.67 MB/sec execute 397 sec latency 44.651 ms 2024-04-27T01:15:15.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:15 smithi135 ceph-mon[50158]: pgmap v148: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 156/1743 objects degraded (8.950%) 2024-04-27T01:15:15.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:15 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:15.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:15 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:15.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:15 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:16.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 228480 10.65 MB/sec execute 398 sec latency 75.430 ms 2024-04-27T01:15:17.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 228879 10.65 MB/sec execute 399 sec latency 44.094 ms 2024-04-27T01:15:17.690 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:17 smithi104 ceph-mon[63925]: pgmap v149: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 408 op/s; 156/1743 objects degraded (8.950%) 2024-04-27T01:15:17.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:17 smithi135 ceph-mon[50158]: pgmap v149: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 408 op/s; 156/1743 objects degraded (8.950%) 2024-04-27T01:15:18.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 229264 10.65 MB/sec execute 400 sec latency 48.843 ms 2024-04-27T01:15:18.730 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:18 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 148/1707 objects degraded (8.670%), 20 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:18.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:18 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 148/1707 objects degraded (8.670%), 20 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:19.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 229654 10.65 MB/sec execute 401 sec latency 51.655 ms 2024-04-27T01:15:19.762 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:19 smithi104 ceph-mon[63925]: pgmap v150: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.6 MiB/s wr, 276 op/s; 148/1707 objects degraded (8.670%) 2024-04-27T01:15:19.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:19 smithi135 ceph-mon[50158]: pgmap v150: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.6 MiB/s wr, 276 op/s; 148/1707 objects degraded (8.670%) 2024-04-27T01:15:20.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 230042 10.65 MB/sec execute 402 sec latency 43.827 ms 2024-04-27T01:15:21.679 INFO:teuthology.orchestra.run.smithi104.stdout: 5 230351 10.65 MB/sec execute 403 sec latency 33.905 ms 2024-04-27T01:15:21.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:21 smithi104 ceph-mon[63925]: pgmap v151: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 327 op/s; 149/1653 objects degraded (9.014%) 2024-04-27T01:15:21.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:21 smithi135 ceph-mon[50158]: pgmap v151: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 327 op/s; 149/1653 objects degraded (9.014%) 2024-04-27T01:15:22.295 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:22 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:15:22.286+0000 7fe9620947c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-27T01:15:22.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 230661 10.66 MB/sec execute 404 sec latency 33.976 ms 2024-04-27T01:15:23.583 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:23 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:15:23.508+0000 7fe9620947c0 -1 osd.2 106 log_to_monitors true 2024-04-27T01:15:23.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 230953 10.65 MB/sec execute 405 sec latency 42.257 ms 2024-04-27T01:15:23.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:23 smithi104 ceph-mon[63925]: pgmap v152: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 461 op/s; 148/1656 objects degraded (8.937%) 2024-04-27T01:15:23.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:23 smithi104 ceph-mon[63925]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:15:23.834 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:15:23 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:15:23.658+0000 7fe9534ec700 -1 osd.2 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:15:23.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:23 smithi135 ceph-mon[50158]: pgmap v152: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.3 MiB/s wr, 461 op/s; 148/1656 objects degraded (8.937%) 2024-04-27T01:15:23.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:23 smithi135 ceph-mon[50158]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:15:24.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231289 10.65 MB/sec execute 406 sec latency 77.321 ms 2024-04-27T01:15:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:24 smithi135 ceph-mon[50158]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:15:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:24 smithi135 ceph-mon[50158]: osdmap e109: 8 total, 7 up, 8 in 2024-04-27T01:15:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:24 smithi135 ceph-mon[50158]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:15:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:24.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:24.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:24 smithi104 ceph-mon[63925]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:15:24.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:24 smithi104 ceph-mon[63925]: osdmap e109: 8 total, 7 up, 8 in 2024-04-27T01:15:24.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:24 smithi104 ceph-mon[63925]: from='osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:15:24.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:24.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:25.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231289 10.62 MB/sec execute 407 sec latency 1077.352 ms 2024-04-27T01:15:25.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: pgmap v154: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 427 op/s; 153/1692 objects degraded (9.043%) 2024-04-27T01:15:25.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975] boot 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 153/1692 objects degraded (9.043%), 20 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:25.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:25 smithi104 ceph-mon[63925]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T01:15:26.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: pgmap v154: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 427 op/s; 153/1692 objects degraded (9.043%) 2024-04-27T01:15:26.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:15:26.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: osd.2 [v2:172.21.15.104:6818/1372092975,v1:172.21.15.104:6819/1372092975] boot 2024-04-27T01:15:26.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T01:15:26.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:15:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 153/1692 objects degraded (9.043%), 20 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:26.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:25 smithi135 ceph-mon[50158]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T01:15:26.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231289 10.59 MB/sec execute 408 sec latency 2077.377 ms 2024-04-27T01:15:27.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231344 10.57 MB/sec execute 409 sec latency 2719.450 ms 2024-04-27T01:15:27.714 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:27 smithi104 ceph-mon[63925]: pgmap v157: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 417 op/s; 153/1692 objects degraded (9.043%) 2024-04-27T01:15:27.714 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:27 smithi104 ceph-mon[63925]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T01:15:27.714 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:27.714 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:27 smithi135 ceph-mon[50158]: pgmap v157: 97 pgs: 20 active+undersized+degraded, 5 active+undersized, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 417 op/s; 153/1692 objects degraded (9.043%) 2024-04-27T01:15:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:27 smithi135 ceph-mon[50158]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T01:15:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:27.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:28.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231571 10.55 MB/sec execute 410 sec latency 217.462 ms 2024-04-27T01:15:29.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231643 10.53 MB/sec execute 411 sec latency 1037.816 ms 2024-04-27T01:15:29.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:29 smithi104 ceph-mon[63925]: pgmap v159: 97 pgs: 81 active+clean, 2 active+undersized, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+undersized+remapped; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s wr, 324 op/s; 117/1641 objects degraded (7.130%); 379 KiB/s, 12 objects/s recovering 2024-04-27T01:15:29.720 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:29 smithi104 ceph-mon[63925]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T01:15:29.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:29 smithi135 ceph-mon[50158]: pgmap v159: 97 pgs: 81 active+clean, 2 active+undersized, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 1 active+undersized+remapped; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s wr, 324 op/s; 117/1641 objects degraded (7.130%); 379 KiB/s, 12 objects/s recovering 2024-04-27T01:15:29.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:29 smithi135 ceph-mon[50158]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T01:15:30.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 231906 10.51 MB/sec execute 412 sec latency 967.699 ms 2024-04-27T01:15:30.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:30 smithi104 ceph-mon[63925]: osdmap e114: 8 total, 8 up, 8 in 2024-04-27T01:15:30.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:30 smithi135 ceph-mon[50158]: osdmap e114: 8 total, 8 up, 8 in 2024-04-27T01:15:31.498 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: pgmap v162: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 MiB/s wr, 348 op/s; 45/1551 objects degraded (2.901%); 673 KiB/s, 24 objects/s recovering 2024-04-27T01:15:31.498 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 45/1551 objects degraded (2.901%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:31.498 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: osdmap e115: 8 total, 8 up, 8 in 2024-04-27T01:15:31.498 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.498 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: pgmap v164: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s wr, 403 op/s; 45/1551 objects degraded (2.901%); 780 KiB/s, 28 objects/s recovering 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: pgmap v165: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 107 op/s; 45/1551 objects degraded (2.901%); 609 KiB/s, 24 objects/s recovering 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.499 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:31 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:31.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 232264 10.51 MB/sec execute 413 sec latency 42.695 ms 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: pgmap v162: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 MiB/s wr, 348 op/s; 45/1551 objects degraded (2.901%); 673 KiB/s, 24 objects/s recovering 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 45/1551 objects degraded (2.901%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: osdmap e115: 8 total, 8 up, 8 in 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: pgmap v164: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s wr, 403 op/s; 45/1551 objects degraded (2.901%); 780 KiB/s, 28 objects/s recovering 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: pgmap v165: 97 pgs: 2 peering, 91 active+clean, 1 active+recovery_wait+degraded, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 107 op/s; 45/1551 objects degraded (2.901%); 609 KiB/s, 24 objects/s recovering 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:31.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:31 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:32.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 232651 10.51 MB/sec execute 414 sec latency 46.678 ms 2024-04-27T01:15:33.015 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:32 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:33.015 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:32 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-27T01:15:33.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:32 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:33.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:32 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-27T01:15:33.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 233148 10.51 MB/sec execute 415 sec latency 37.676 ms 2024-04-27T01:15:34.041 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:33 smithi104 ceph-mon[63925]: pgmap v166: 97 pgs: 2 peering, 93 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s wr, 70 op/s; 4/1536 objects degraded (0.260%); 735 KiB/s, 22 objects/s recovering 2024-04-27T01:15:34.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:33 smithi135 ceph-mon[50158]: pgmap v166: 97 pgs: 2 peering, 93 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 MiB/s wr, 70 op/s; 4/1536 objects degraded (0.260%); 735 KiB/s, 22 objects/s recovering 2024-04-27T01:15:34.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 233605 10.52 MB/sec execute 416 sec latency 29.795 ms 2024-04-27T01:15:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:34 smithi104 ceph-mon[63925]: pgmap v167: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.2 MiB/s wr, 394 op/s; 1.1 MiB/s, 9 objects/s recovering 2024-04-27T01:15:35.083 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:34 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1536 objects degraded (0.260%), 1 pg degraded) 2024-04-27T01:15:35.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:34 smithi135 ceph-mon[50158]: pgmap v167: 97 pgs: 97 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.2 MiB/s wr, 394 op/s; 1.1 MiB/s, 9 objects/s recovering 2024-04-27T01:15:35.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:34 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1536 objects degraded (0.260%), 1 pg degraded) 2024-04-27T01:15:35.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 234047 10.54 MB/sec execute 417 sec latency 38.674 ms 2024-04-27T01:15:36.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 234392 10.54 MB/sec execute 418 sec latency 38.619 ms 2024-04-27T01:15:37.066 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:15:37.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 234689 10.53 MB/sec execute 419 sec latency 72.366 ms 2024-04-27T01:15:37.887 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:37 smithi104 ceph-mon[63925]: pgmap v168: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 352 op/s; 939 KiB/s, 7 objects/s recovering 2024-04-27T01:15:38.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:37 smithi135 ceph-mon[50158]: pgmap v168: 97 pgs: 97 active+clean; 325 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 352 op/s; 939 KiB/s, 7 objects/s recovering 2024-04-27T01:15:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:15:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (6m) 13s ago 15m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:15:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (2m) 13s ago 15m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:15:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (2m) 2m ago 14m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:15:38.114 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 13s ago 15m - - 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (9m) 13s ago 9m 40.8M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (9m) 2m ago 9m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (5m) 13s ago 16m 506M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (4m) 2m ago 14m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (4m) 13s ago 16m 49.5M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (3m) 2m ago 14m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (9m) 13s ago 9m 275M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (9m) 2m ago 9m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 13s ago 15m 21.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (5m) 2m ago 14m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (119s) 13s ago 14m 301M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (72s) 13s ago 13m 244M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (23s) 13s ago 13m 12.6M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:15:38.115 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (12m) 13s ago 12m 418M 1936M 17.2.0 e1d6a67b021e 88de12aae775 2024-04-27T01:15:38.116 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (12m) 2m ago 12m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:15:38.116 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (11m) 2m ago 11m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:15:38.116 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (11m) 2m ago 11m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:15:38.116 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (11m) 2m ago 11m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:15:38.116 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (4m) 13s ago 14m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:15:38.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 234950 10.52 MB/sec execute 420 sec latency 49.104 ms 2024-04-27T01:15:38.778 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 3 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:15:38.779 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:15:38.780 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:15:38.780 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-27T01:15:38.780 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 7 2024-04-27T01:15:38.780 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:15:38.780 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:15:38.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:38 smithi104 ceph-mon[63925]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:38.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:38 smithi104 ceph-mon[63925]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:38.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:38 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3142150168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:38 smithi135 ceph-mon[50158]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:38 smithi135 ceph-mon[50158]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:39.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:38 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3142150168' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "9/23 daemons upgraded", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:15:39.382 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:15:39.383 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:15:39.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 235289 10.51 MB/sec execute 421 sec latency 48.655 ms 2024-04-27T01:15:39.939 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:39 smithi104 ceph-mon[63925]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:39.939 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:39 smithi104 ceph-mon[63925]: pgmap v169: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.5 MiB/s wr, 575 op/s; 716 KiB/s, 5 objects/s recovering 2024-04-27T01:15:40.033 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:15:40.034 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:15:40.034 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:15:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:39 smithi135 ceph-mon[50158]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:39 smithi135 ceph-mon[50158]: pgmap v169: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.5 MiB/s wr, 575 op/s; 716 KiB/s, 5 objects/s recovering 2024-04-27T01:15:40.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 235620 10.50 MB/sec execute 422 sec latency 39.694 ms 2024-04-27T01:15:41.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:40 smithi104 ceph-mon[63925]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:41.031 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:40 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/578541078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:15:41.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:40 smithi135 ceph-mon[50158]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:15:41.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:40 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/578541078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:15:41.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 236022 10.50 MB/sec execute 423 sec latency 59.106 ms 2024-04-27T01:15:42.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:41 smithi104 ceph-mon[63925]: pgmap v170: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 490 op/s; 594 KiB/s, 4 objects/s recovering 2024-04-27T01:15:42.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:41 smithi135 ceph-mon[50158]: pgmap v170: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 490 op/s; 594 KiB/s, 4 objects/s recovering 2024-04-27T01:15:42.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 236457 10.50 MB/sec execute 424 sec latency 43.082 ms 2024-04-27T01:15:43.568 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:43.568 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:43.568 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:43 smithi104 ceph-mon[63925]: pgmap v171: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 480 op/s; 495 KiB/s, 3 objects/s recovering 2024-04-27T01:15:43.569 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:43.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 236838 10.50 MB/sec execute 425 sec latency 44.087 ms 2024-04-27T01:15:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:43 smithi135 ceph-mon[50158]: pgmap v171: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 480 op/s; 495 KiB/s, 3 objects/s recovering 2024-04-27T01:15:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:44.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 237196 10.51 MB/sec execute 426 sec latency 55.121 ms 2024-04-27T01:15:45.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 237581 10.52 MB/sec execute 427 sec latency 31.175 ms 2024-04-27T01:15:45.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:45 smithi104 ceph-mon[63925]: pgmap v172: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 594 op/s; 349 KiB/s, 0 objects/s recovering 2024-04-27T01:15:46.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:45 smithi135 ceph-mon[50158]: pgmap v172: 97 pgs: 97 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 594 op/s; 349 KiB/s, 0 objects/s recovering 2024-04-27T01:15:46.680 INFO:teuthology.orchestra.run.smithi104.stdout: 5 237844 10.51 MB/sec execute 428 sec latency 57.266 ms 2024-04-27T01:15:46.889 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:47.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 238119 10.51 MB/sec execute 429 sec latency 58.701 ms 2024-04-27T01:15:47.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:47 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:47.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:47 smithi104 ceph-mon[63925]: Upgrade: osd.3 is safe to restart 2024-04-27T01:15:47.902 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:47 smithi104 ceph-mon[63925]: pgmap v173: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 476 op/s 2024-04-27T01:15:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:47 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-27T01:15:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:47 smithi135 ceph-mon[50158]: Upgrade: osd.3 is safe to restart 2024-04-27T01:15:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:47 smithi135 ceph-mon[50158]: pgmap v173: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.9 MiB/s wr, 476 op/s 2024-04-27T01:15:48.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 238340 10.49 MB/sec execute 430 sec latency 52.720 ms 2024-04-27T01:15:49.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 238590 10.48 MB/sec execute 431 sec latency 72.933 ms 2024-04-27T01:15:49.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:49 smithi104 ceph-mon[63925]: pgmap v174: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.3 MiB/s wr, 590 op/s 2024-04-27T01:15:49.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:49.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:15:49.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:49 smithi135 ceph-mon[50158]: pgmap v174: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.3 MiB/s wr, 590 op/s 2024-04-27T01:15:50.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:15:50.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:50.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 238842 10.47 MB/sec execute 432 sec latency 55.959 ms 2024-04-27T01:15:50.993 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:50 smithi104 ceph-mon[63925]: Upgrade: Updating osd.3 2024-04-27T01:15:51.015 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:50 smithi104 ceph-mon[63925]: Deploying daemon osd.3 on smithi104 2024-04-27T01:15:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:50 smithi135 ceph-mon[50158]: Upgrade: Updating osd.3 2024-04-27T01:15:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:50 smithi135 ceph-mon[50158]: Deploying daemon osd.3 on smithi104 2024-04-27T01:15:51.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 239057 10.45 MB/sec execute 433 sec latency 75.849 ms 2024-04-27T01:15:52.016 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:51 smithi104 ceph-mon[63925]: pgmap v175: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 386 op/s 2024-04-27T01:15:52.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:51 smithi135 ceph-mon[50158]: pgmap v175: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 386 op/s 2024-04-27T01:15:52.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 239276 10.43 MB/sec execute 434 sec latency 64.591 ms 2024-04-27T01:15:53.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 239562 10.43 MB/sec execute 435 sec latency 106.694 ms 2024-04-27T01:15:53.840 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:53 smithi104 systemd[1]: Stopping Ceph osd.3 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:15:54.102 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:53 smithi104 ceph-mon[63925]: pgmap v176: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 449 op/s 2024-04-27T01:15:54.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:53 smithi135 ceph-mon[50158]: pgmap v176: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 449 op/s 2024-04-27T01:15:54.375 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:54 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[42823]: 2024-04-27T01:15:54.164+0000 7f5e5c607700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:15:54.376 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:54 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[42823]: 2024-04-27T01:15:54.164+0000 7f5e5c607700 -1 osd.3 115 *** Got signal Terminated *** 2024-04-27T01:15:54.376 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:54 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[42823]: 2024-04-27T01:15:54.164+0000 7f5e5c607700 -1 osd.3 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:15:54.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 239840 10.42 MB/sec execute 436 sec latency 67.623 ms 2024-04-27T01:15:54.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:54 smithi104 ceph-mon[63925]: osd.3 marked itself down and dead 2024-04-27T01:15:55.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:54 smithi135 ceph-mon[50158]: osd.3 marked itself down and dead 2024-04-27T01:15:55.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 239979 10.40 MB/sec execute 437 sec latency 808.244 ms 2024-04-27T01:15:55.853 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:55 smithi104 podman[77322]: 2024-04-27 01:15:55.547004593 +0000 UTC m=+1.549745684 container died 88de12aae775a5e2251dad12a1e328df4e1a085e31285aefd5fe290deb15ae5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, ceph=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, version=8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, vendor=Red Hat, Inc., release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container) 2024-04-27T01:15:56.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:55 smithi104 ceph-mon[63925]: pgmap v177: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.1 MiB/s wr, 509 op/s 2024-04-27T01:15:56.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:55 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:15:56.103 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:55 smithi104 ceph-mon[63925]: osdmap e116: 8 total, 7 up, 8 in 2024-04-27T01:15:56.103 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:56 smithi104 podman[77322]: 2024-04-27 01:15:56.106210863 +0000 UTC m=+2.108951959 container cleanup 88de12aae775a5e2251dad12a1e328df4e1a085e31285aefd5fe290deb15ae5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream) 2024-04-27T01:15:56.103 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:56 smithi104 bash[77322]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3 2024-04-27T01:15:56.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:55 smithi135 ceph-mon[50158]: pgmap v177: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.1 MiB/s wr, 509 op/s 2024-04-27T01:15:56.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:55 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:15:56.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:55 smithi135 ceph-mon[50158]: osdmap e116: 8 total, 7 up, 8 in 2024-04-27T01:15:56.620 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:56 smithi104 podman[77373]: 2024-04-27 01:15:56.466710596 +0000 UTC m=+0.911966090 container remove 88de12aae775a5e2251dad12a1e328df4e1a085e31285aefd5fe290deb15ae5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, RELEASE=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc.) 2024-04-27T01:15:56.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 240313 10.41 MB/sec execute 438 sec latency 1048.770 ms 2024-04-27T01:15:57.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:56 smithi104 ceph-mon[63925]: osdmap e117: 8 total, 7 up, 8 in 2024-04-27T01:15:57.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:56 smithi104 ceph-mon[63925]: pgmap v180: 97 pgs: 3 active+undersized, 6 active+undersized+degraded, 12 stale+active+clean, 76 active+clean; 329 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 385 op/s; 32/1530 objects degraded (2.092%) 2024-04-27T01:15:57.134 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:56 smithi104 podman[77452]: 2024-04-27 01:15:56.80462724 +0000 UTC m=+0.021156186 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:15:57.134 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:56 smithi104 podman[77452]: 2024-04-27 01:15:56.93600506 +0000 UTC m=+0.152533997 container create 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T01:15:57.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:56 smithi135 ceph-mon[50158]: osdmap e117: 8 total, 7 up, 8 in 2024-04-27T01:15:57.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:56 smithi135 ceph-mon[50158]: pgmap v180: 97 pgs: 3 active+undersized, 6 active+undersized+degraded, 12 stale+active+clean, 76 active+clean; 329 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 385 op/s; 32/1530 objects degraded (2.092%) 2024-04-27T01:15:57.393 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:57 smithi104 podman[77452]: 2024-04-27 01:15:57.28871821 +0000 UTC m=+0.505247159 container init 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:15:57.393 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:57 smithi104 podman[77452]: 2024-04-27 01:15:57.300841044 +0000 UTC m=+0.517369986 container start 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-27T01:15:57.393 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:57 smithi104 podman[77452]: 2024-04-27 01:15:57.397138896 +0000 UTC m=+0.613667833 container attach 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:15:57.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 240714 10.41 MB/sec execute 439 sec latency 31.731 ms 2024-04-27T01:15:57.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:57 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 32/1530 objects degraded (2.092%), 6 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:57.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:57.898 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:57.898 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:57 smithi104 podman[77478]: 2024-04-27 01:15:57.782600862 +0000 UTC m=+0.024133519 container died 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:15:58.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:57 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 32/1530 objects degraded (2.092%), 6 pgs degraded (PG_DEGRADED) 2024-04-27T01:15:58.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:15:58.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:58.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 241225 10.43 MB/sec execute 440 sec latency 34.513 ms 2024-04-27T01:15:58.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:15:58 smithi104 ceph-mon[63925]: pgmap v181: 97 pgs: 12 active+undersized, 20 active+undersized+degraded, 3 stale+active+clean, 62 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 463 op/s; 145/1614 objects degraded (8.984%) 2024-04-27T01:15:58.907 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:58 smithi104 podman[77478]: 2024-04-27 01:15:58.71985606 +0000 UTC m=+0.961388717 container remove 55bcb41ecd6c41fd84eb8405cf47e23c2f96ab523459fa45fe373001e4811c4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:15:58.907 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:58 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-27T01:15:58.907 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:58 smithi104 systemd[1]: Stopped Ceph osd.3 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:15:58.907 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:58 smithi104 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.3.service: Consumed 4min 51.487s CPU time. 2024-04-27T01:15:59.158 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:59 smithi104 systemd[1]: Starting Ceph osd.3 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:15:59.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:15:58 smithi135 ceph-mon[50158]: pgmap v181: 97 pgs: 12 active+undersized, 20 active+undersized+degraded, 3 stale+active+clean, 62 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 463 op/s; 145/1614 objects degraded (8.984%) 2024-04-27T01:15:59.409 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:59 smithi104 podman[77576]: 2024-04-27 01:15:59.272816166 +0000 UTC m=+0.026116013 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:15:59.410 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:59 smithi104 podman[77576]: 2024-04-27 01:15:59.416886992 +0000 UTC m=+0.170186819 container create e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-27T01:15:59.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 241713 10.44 MB/sec execute 441 sec latency 23.962 ms 2024-04-27T01:15:59.915 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:59 smithi104 podman[77576]: 2024-04-27 01:15:59.838130644 +0000 UTC m=+0.591430484 container init e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-27T01:15:59.915 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:15:59 smithi104 podman[77576]: 2024-04-27 01:15:59.84409676 +0000 UTC m=+0.597396586 container start e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:16:00.171 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 podman[77576]: 2024-04-27 01:16:00.02472093 +0000 UTC m=+0.778020759 container attach e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T01:16:00.674 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 bash[77576]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate[77590]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-27T01:16:00.675 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:00 smithi104 podman[77576]: 2024-04-27 01:16:00.525620362 +0000 UTC m=+1.278920200 container died e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-27T01:16:00.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 242136 10.43 MB/sec execute 442 sec latency 76.071 ms 2024-04-27T01:16:01.680 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:01 smithi104 podman[77720]: 2024-04-27 01:16:01.640067281 +0000 UTC m=+1.108278592 container remove e453474cb46ee54426e77ead3ae0745463ceef43e8ddc694f6713d597410e258 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb) 2024-04-27T01:16:01.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 242553 10.43 MB/sec execute 443 sec latency 44.691 ms 2024-04-27T01:16:01.941 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:01 smithi104 ceph-mon[63925]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 379 op/s; 221/1695 objects degraded (13.038%) 2024-04-27T01:16:01.941 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:01 smithi104 podman[77777]: 2024-04-27 01:16:01.803153272 +0000 UTC m=+0.024136992 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:16:01.941 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:01 smithi104 podman[77777]: 2024-04-27 01:16:01.924637472 +0000 UTC m=+0.145621167 container create af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-27T01:16:02.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:01 smithi135 ceph-mon[50158]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 379 op/s; 221/1695 objects degraded (13.038%) 2024-04-27T01:16:02.444 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 podman[77777]: 2024-04-27 01:16:02.305605212 +0000 UTC m=+0.526588915 container init af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-27T01:16:02.444 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 podman[77777]: 2024-04-27 01:16:02.314971509 +0000 UTC m=+0.535955197 container start af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-27T01:16:02.444 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-osd[77795]: -- 172.21.15.104:0/2591082402 <== mon.0 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56069501d680 con 0x560694fe6c00 2024-04-27T01:16:02.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 242945 10.42 MB/sec execute 444 sec latency 50.559 ms 2024-04-27T01:16:02.694 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 bash[77777]: af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 2024-04-27T01:16:02.694 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 systemd[1]: Started Ceph osd.3 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:16:02.947 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:16:02.896+0000 7faf64e117c0 -1 Falling back to public interface 2024-04-27T01:16:02.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 221/1695 objects degraded (13.038%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:02.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:02.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:02.947 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:03.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:02 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 221/1695 objects degraded (13.038%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:03.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:03.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:03.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:03.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 243332 10.42 MB/sec execute 445 sec latency 49.802 ms 2024-04-27T01:16:03.968 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:03 smithi104 ceph-mon[63925]: pgmap v183: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s wr, 294 op/s; 224/1737 objects degraded (12.896%) 2024-04-27T01:16:04.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:03 smithi135 ceph-mon[50158]: pgmap v183: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.5 MiB/s wr, 294 op/s; 224/1737 objects degraded (12.896%) 2024-04-27T01:16:04.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 243812 10.43 MB/sec execute 446 sec latency 34.848 ms 2024-04-27T01:16:05.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 244226 10.43 MB/sec execute 447 sec latency 37.425 ms 2024-04-27T01:16:06.011 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:05 smithi104 ceph-mon[63925]: pgmap v184: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.5 MiB/s wr, 558 op/s; 203/1686 objects degraded (12.040%) 2024-04-27T01:16:06.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:05 smithi135 ceph-mon[50158]: pgmap v184: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 5.5 MiB/s wr, 558 op/s; 203/1686 objects degraded (12.040%) 2024-04-27T01:16:06.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 244506 10.44 MB/sec execute 448 sec latency 52.010 ms 2024-04-27T01:16:07.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 244818 10.44 MB/sec execute 449 sec latency 49.573 ms 2024-04-27T01:16:08.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:07 smithi104 ceph-mon[63925]: pgmap v185: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.3 MiB/s wr, 520 op/s; 199/1656 objects degraded (12.017%) 2024-04-27T01:16:08.069 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:07 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 199/1656 objects degraded (12.017%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:08.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:07 smithi135 ceph-mon[50158]: pgmap v185: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.3 MiB/s wr, 520 op/s; 199/1656 objects degraded (12.017%) 2024-04-27T01:16:08.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:07 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 199/1656 objects degraded (12.017%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:08.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 245071 10.43 MB/sec execute 450 sec latency 60.519 ms 2024-04-27T01:16:09.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 245293 10.42 MB/sec execute 451 sec latency 58.141 ms 2024-04-27T01:16:09.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:09 smithi104 ceph-mon[63925]: pgmap v186: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.3 MiB/s wr, 604 op/s; 202/1677 objects degraded (12.045%) 2024-04-27T01:16:09.888 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:09 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:16:09.670+0000 7faf64e117c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-27T01:16:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:09 smithi135 ceph-mon[50158]: pgmap v186: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.3 MiB/s wr, 604 op/s; 202/1677 objects degraded (12.045%) 2024-04-27T01:16:10.620 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:16:10.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 245510 10.41 MB/sec execute 452 sec latency 48.011 ms 2024-04-27T01:16:11.418 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:11 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:16:11.294+0000 7faf64e117c0 -1 osd.3 115 log_to_monitors true 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (7m) 47s ago 15m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (3m) 47s ago 15m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (3m) 3m ago 15m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 47s ago 15m - - 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (10m) 47s ago 10m 40.8M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (10m) 3m ago 10m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:16:11.623 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (5m) 47s ago 17m 506M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (4m) 3m ago 15m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (4m) 47s ago 17m 49.5M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (4m) 3m ago 15m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (9m) 47s ago 9m 275M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (9m) 3m ago 9m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 47s ago 15m 21.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 3m ago 15m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (2m) 47s ago 14m 301M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (105s) 47s ago 14m 244M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (57s) 47s ago 13m 12.6M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 starting - - - 1936M 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (12m) 3m ago 12m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (12m) 3m ago 12m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:16:11.624 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (11m) 3m ago 11m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:16:11.625 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (11m) 3m ago 11m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:16:11.625 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (5m) 47s ago 14m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:16:11.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 245742 10.39 MB/sec execute 453 sec latency 71.509 ms 2024-04-27T01:16:11.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:11 smithi104 ceph-mon[63925]: from='client.44247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:11.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:11 smithi104 ceph-mon[63925]: pgmap v187: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.5 MiB/s wr, 559 op/s; 204/1707 objects degraded (11.951%) 2024-04-27T01:16:11.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:11 smithi104 ceph-mon[63925]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:16:11.941 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:16:11 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:16:11.922+0000 7faf56269700 -1 osd.3 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:16:12.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:11 smithi135 ceph-mon[50158]: from='client.44247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:12.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:11 smithi135 ceph-mon[50158]: pgmap v187: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.5 MiB/s wr, 559 op/s; 204/1707 objects degraded (11.951%) 2024-04-27T01:16:12.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:11 smithi135 ceph-mon[50158]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T01:16:12.212 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 3 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 7 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:16:12.213 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:16:12.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246042 10.38 MB/sec execute 454 sec latency 61.831 ms 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "10/23 daemons upgraded", 2024-04-27T01:16:12.736 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:16:12.737 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:16:12.737 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: osdmap e118: 8 total, 7 up, 8 in 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 204/1707 objects degraded (11.951%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3637251196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:12.960 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:13.277 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 202/1698 objects degraded (11.896%), 27 pgs degraded 2024-04-27T01:16:13.277 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:16:13.277 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 202/1698 objects degraded (11.896%), 27 pgs degraded 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-04-27T01:16:13.278 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-27T01:16:13.279 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-27T01:16:13.281 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-04-27T01:16:13.282 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-27T01:16:13.282 INFO:teuthology.orchestra.run.smithi104.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-27T01:16:13.282 INFO:teuthology.orchestra.run.smithi104.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-04-27T01:16:13.282 INFO:teuthology.orchestra.run.smithi104.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: osdmap e118: 8 total, 7 up, 8 in 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 204/1707 objects degraded (11.951%), 27 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3637251196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:13.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:13.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246128 10.36 MB/sec execute 455 sec latency 779.674 ms 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: pgmap v189: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 568 op/s; 202/1698 objects degraded (11.896%) 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231] boot 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:16:14.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/4046636990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:16:14.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:14.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: pgmap v189: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.6 MiB/s wr, 568 op/s; 202/1698 objects degraded (11.896%) 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: osd.3 [v2:172.21.15.104:6826/4145233231,v1:172.21.15.104:6827/4145233231] boot 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/4046636990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:16:14.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:14.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:14.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246129 10.34 MB/sec execute 456 sec latency 1779.697 ms 2024-04-27T01:16:15.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:14 smithi104 ceph-mon[63925]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T01:16:15.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:14 smithi104 ceph-mon[63925]: pgmap v192: 97 pgs: 1 unknown, 19 peering, 5 active+undersized, 16 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.9 MiB/s wr, 316 op/s; 102/1629 objects degraded (6.262%) 2024-04-27T01:16:15.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:14 smithi135 ceph-mon[50158]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T01:16:15.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:14 smithi135 ceph-mon[50158]: pgmap v192: 97 pgs: 1 unknown, 19 peering, 5 active+undersized, 16 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.9 MiB/s wr, 316 op/s; 102/1629 objects degraded (6.262%) 2024-04-27T01:16:15.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246188 10.32 MB/sec execute 457 sec latency 2461.167 ms 2024-04-27T01:16:16.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:15 smithi104 ceph-mon[63925]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T01:16:16.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:15 smithi135 ceph-mon[50158]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T01:16:16.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246469 10.31 MB/sec execute 458 sec latency 104.611 ms 2024-04-27T01:16:17.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:16 smithi104 ceph-mon[63925]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T01:16:17.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:16 smithi104 ceph-mon[63925]: pgmap v195: 97 pgs: 1 unknown, 20 peering, 4 active+undersized, 1 active+undersized+remapped, 11 active+undersized+degraded, 60 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 427 op/s; 74/1617 objects degraded (4.576%); 942 KiB/s, 9 objects/s recovering 2024-04-27T01:16:17.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:16 smithi135 ceph-mon[50158]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T01:16:17.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:16 smithi135 ceph-mon[50158]: pgmap v195: 97 pgs: 1 unknown, 20 peering, 4 active+undersized, 1 active+undersized+remapped, 11 active+undersized+degraded, 60 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 427 op/s; 74/1617 objects degraded (4.576%); 942 KiB/s, 9 objects/s recovering 2024-04-27T01:16:17.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 246798 10.31 MB/sec execute 459 sec latency 733.678 ms 2024-04-27T01:16:18.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:17 smithi104 ceph-mon[63925]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T01:16:18.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:17 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 74/1617 objects degraded (4.576%), 11 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:18.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:17 smithi135 ceph-mon[50158]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T01:16:18.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:17 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 74/1617 objects degraded (4.576%), 11 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:18.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 247092 10.30 MB/sec execute 460 sec latency 995.005 ms 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: pgmap v198: 97 pgs: 1 unknown, 21 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 73 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 457 KiB/s wr, 44 op/s; 14/1611 objects degraded (0.869%); 1.2 MiB/s, 36 objects/s recovering 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: pgmap v199: 97 pgs: 1 unknown, 21 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 73 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 470 KiB/s wr, 45 op/s; 14/1611 objects degraded (0.869%); 1.2 MiB/s, 37 objects/s recovering 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:18.931 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:18.932 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:18 smithi104 ceph-mon[63925]: Upgrade: 1 pgs have unknown state; cannot draw any conclusions 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: pgmap v198: 97 pgs: 1 unknown, 21 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 73 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 457 KiB/s wr, 44 op/s; 14/1611 objects degraded (0.869%); 1.2 MiB/s, 36 objects/s recovering 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: pgmap v199: 97 pgs: 1 unknown, 21 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 73 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 470 KiB/s wr, 45 op/s; 14/1611 objects degraded (0.869%); 1.2 MiB/s, 37 objects/s recovering 2024-04-27T01:16:19.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:19.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:18 smithi135 ceph-mon[50158]: Upgrade: 1 pgs have unknown state; cannot draw any conclusions 2024-04-27T01:16:19.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 247440 10.30 MB/sec execute 461 sec latency 75.202 ms 2024-04-27T01:16:20.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 247694 10.30 MB/sec execute 462 sec latency 39.862 ms 2024-04-27T01:16:21.240 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:21 smithi104 ceph-mon[63925]: pgmap v200: 97 pgs: 2 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 93 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.4 MiB/s wr, 325 op/s; 14/1557 objects degraded (0.899%); 3.1 MiB/s, 59 objects/s recovering 2024-04-27T01:16:21.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:21 smithi135 ceph-mon[50158]: pgmap v200: 97 pgs: 2 peering, 1 active+undersized+remapped, 1 active+undersized+degraded, 93 active+clean; 326 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.4 MiB/s wr, 325 op/s; 14/1557 objects degraded (0.899%); 3.1 MiB/s, 59 objects/s recovering 2024-04-27T01:16:21.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 248085 10.31 MB/sec execute 463 sec latency 38.874 ms 2024-04-27T01:16:22.256 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:22 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1557 objects degraded (0.899%), 1 pg degraded) 2024-04-27T01:16:22.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:22 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1557 objects degraded (0.899%), 1 pg degraded) 2024-04-27T01:16:22.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 248406 10.31 MB/sec execute 464 sec latency 46.814 ms 2024-04-27T01:16:23.291 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:23 smithi104 ceph-mon[63925]: pgmap v201: 97 pgs: 97 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.4 MiB/s wr, 300 op/s; 1.7 MiB/s, 43 objects/s recovering 2024-04-27T01:16:23.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:23 smithi135 ceph-mon[50158]: pgmap v201: 97 pgs: 97 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.4 MiB/s wr, 300 op/s; 1.7 MiB/s, 43 objects/s recovering 2024-04-27T01:16:23.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 248674 10.31 MB/sec execute 465 sec latency 49.894 ms 2024-04-27T01:16:24.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 248861 10.29 MB/sec execute 466 sec latency 40.439 ms 2024-04-27T01:16:25.315 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:25 smithi104 ceph-mon[63925]: pgmap v202: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.4 MiB/s wr, 496 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-04-27T01:16:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:25 smithi135 ceph-mon[50158]: pgmap v202: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.4 MiB/s wr, 496 op/s; 1.2 MiB/s, 21 objects/s recovering 2024-04-27T01:16:25.681 INFO:teuthology.orchestra.run.smithi104.stdout: 5 249082 10.28 MB/sec execute 467 sec latency 76.919 ms 2024-04-27T01:16:26.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 249343 10.26 MB/sec execute 468 sec latency 69.510 ms 2024-04-27T01:16:27.344 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:27 smithi104 ceph-mon[63925]: pgmap v203: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.7 MiB/s wr, 435 op/s; 1.1 MiB/s, 18 objects/s recovering 2024-04-27T01:16:27.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:27 smithi135 ceph-mon[50158]: pgmap v203: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.7 MiB/s wr, 435 op/s; 1.1 MiB/s, 18 objects/s recovering 2024-04-27T01:16:27.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 249559 10.25 MB/sec execute 469 sec latency 65.120 ms 2024-04-27T01:16:28.618 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:28.618 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:28.619 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:28.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 249811 10.24 MB/sec execute 470 sec latency 55.219 ms 2024-04-27T01:16:28.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:28.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:28.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:29.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 250282 10.25 MB/sec execute 471 sec latency 35.675 ms 2024-04-27T01:16:29.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:29 smithi104 ceph-mon[63925]: pgmap v204: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 369 op/s; 900 KiB/s, 15 objects/s recovering 2024-04-27T01:16:30.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:29 smithi135 ceph-mon[50158]: pgmap v204: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.2 MiB/s wr, 369 op/s; 900 KiB/s, 15 objects/s recovering 2024-04-27T01:16:30.682 INFO:teuthology.orchestra.run.smithi104.stdout: 5 250789 10.24 MB/sec execute 472 sec latency 52.081 ms 2024-04-27T01:16:31.684 INFO:teuthology.orchestra.run.smithi104.stdout: 5 251122 10.25 MB/sec execute 473 sec latency 35.341 ms 2024-04-27T01:16:31.921 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:31 smithi104 ceph-mon[63925]: pgmap v205: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 750 KiB/s, 12 objects/s recovering 2024-04-27T01:16:32.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:31 smithi135 ceph-mon[50158]: pgmap v205: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 750 KiB/s, 12 objects/s recovering 2024-04-27T01:16:32.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 251528 10.26 MB/sec execute 474 sec latency 44.377 ms 2024-04-27T01:16:33.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 251899 10.26 MB/sec execute 475 sec latency 45.252 ms 2024-04-27T01:16:33.987 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:33 smithi104 ceph-mon[63925]: pgmap v206: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.5 MiB/s wr, 378 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:16:33.987 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:34.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:33 smithi135 ceph-mon[50158]: pgmap v206: 97 pgs: 97 active+clean; 331 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.5 MiB/s wr, 378 op/s; 0 B/s, 1 objects/s recovering 2024-04-27T01:16:34.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:34.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 252220 10.27 MB/sec execute 476 sec latency 37.245 ms 2024-04-27T01:16:35.007 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:34 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:35.007 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:34 smithi104 ceph-mon[63925]: Upgrade: osd.4 is safe to restart 2024-04-27T01:16:35.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:34 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-27T01:16:35.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:34 smithi135 ceph-mon[50158]: Upgrade: osd.4 is safe to restart 2024-04-27T01:16:35.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 252432 10.26 MB/sec execute 477 sec latency 49.105 ms 2024-04-27T01:16:35.918 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:35 smithi135 ceph-mon[50158]: pgmap v207: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 512 op/s 2024-04-27T01:16:35.918 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:35 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:35.918 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:35 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:16:35.918 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:35 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:36.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:35 smithi104 ceph-mon[63925]: pgmap v207: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.3 MiB/s wr, 512 op/s 2024-04-27T01:16:36.017 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:35 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:36.018 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:35 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:16:36.018 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:35 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:36.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 252683 10.25 MB/sec execute 478 sec latency 76.041 ms 2024-04-27T01:16:37.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:36 smithi104 ceph-mon[63925]: Upgrade: Updating osd.4 2024-04-27T01:16:37.035 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:36 smithi104 ceph-mon[63925]: Deploying daemon osd.4 on smithi135 2024-04-27T01:16:37.052 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:36 smithi135 ceph-mon[50158]: Upgrade: Updating osd.4 2024-04-27T01:16:37.052 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:36 smithi135 ceph-mon[50158]: Deploying daemon osd.4 on smithi135 2024-04-27T01:16:37.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 252962 10.23 MB/sec execute 479 sec latency 54.702 ms 2024-04-27T01:16:38.080 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:37 smithi104 ceph-mon[63925]: pgmap v208: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 342 op/s 2024-04-27T01:16:38.108 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:37 smithi135 ceph-mon[50158]: pgmap v208: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.5 MiB/s wr, 342 op/s 2024-04-27T01:16:38.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 253166 10.22 MB/sec execute 480 sec latency 62.312 ms 2024-04-27T01:16:38.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:38 smithi135 systemd[1]: Stopping Ceph osd.4 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:16:39.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:38 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[28316]: 2024-04-27T01:16:38.899+0000 7f183e1d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:16:39.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:38 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[28316]: 2024-04-27T01:16:38.899+0000 7f183e1d1700 -1 osd.4 124 *** Got signal Terminated *** 2024-04-27T01:16:39.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:38 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[28316]: 2024-04-27T01:16:38.899+0000 7f183e1d1700 -1 osd.4 124 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:16:39.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 253396 10.21 MB/sec execute 481 sec latency 55.265 ms 2024-04-27T01:16:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:39 smithi135 ceph-mon[50158]: pgmap v209: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-27T01:16:40.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:39 smithi135 ceph-mon[50158]: osd.4 marked itself down and dead 2024-04-27T01:16:40.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:39 smithi104 ceph-mon[63925]: pgmap v209: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-27T01:16:40.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:39 smithi104 ceph-mon[63925]: osd.4 marked itself down and dead 2024-04-27T01:16:40.606 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:40 smithi135 podman[54642]: 2024-04-27 01:16:40.340303306 +0000 UTC m=+1.559357931 container died 1c30c1b7d02bf7d8e90201b1856557f5236dc8ab9f55cef434873bdc53ff5f78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, version=8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-27T01:16:40.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 253475 10.19 MB/sec execute 482 sec latency 838.163 ms 2024-04-27T01:16:40.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:40 smithi135 podman[54642]: 2024-04-27 01:16:40.708476362 +0000 UTC m=+1.927530980 container cleanup 1c30c1b7d02bf7d8e90201b1856557f5236dc8ab9f55cef434873bdc53ff5f78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_BRANCH=HEAD) 2024-04-27T01:16:40.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:40 smithi135 bash[54642]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4 2024-04-27T01:16:41.117 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:40 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:16:41.118 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:40 smithi104 ceph-mon[63925]: osdmap e125: 8 total, 7 up, 8 in 2024-04-27T01:16:41.118 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:40 smithi104 ceph-mon[63925]: osdmap e126: 8 total, 7 up, 8 in 2024-04-27T01:16:41.359 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:40 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:16:41.360 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:40 smithi135 ceph-mon[50158]: osdmap e125: 8 total, 7 up, 8 in 2024-04-27T01:16:41.360 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:40 smithi135 ceph-mon[50158]: osdmap e126: 8 total, 7 up, 8 in 2024-04-27T01:16:41.360 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54656]: 2024-04-27 01:16:41.010085407 +0000 UTC m=+0.665716139 container remove 1c30c1b7d02bf7d8e90201b1856557f5236dc8ab9f55cef434873bdc53ff5f78 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vendor=Red Hat, Inc., architecture=x86_64) 2024-04-27T01:16:41.613 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.276876492 +0000 UTC m=+0.020786792 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:16:41.613 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.402770957 +0000 UTC m=+0.146681245 container create 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T01:16:41.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 253790 10.19 MB/sec execute 483 sec latency 1034.152 ms 2024-04-27T01:16:41.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:41 smithi104 ceph-mon[63925]: pgmap v211: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 494 op/s 2024-04-27T01:16:41.986 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:41 smithi135 ceph-mon[50158]: pgmap v211: 97 pgs: 14 stale+active+clean, 83 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 494 op/s 2024-04-27T01:16:41.986 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.639203313 +0000 UTC m=+0.383113606 container init 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-27T01:16:41.986 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.643300777 +0000 UTC m=+0.387211067 container start 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-27T01:16:41.986 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.731036644 +0000 UTC m=+0.474946935 container attach 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-27T01:16:42.297 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:41 smithi135 podman[54751]: 2024-04-27 01:16:41.999264333 +0000 UTC m=+0.743174635 container died 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-27T01:16:42.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 254180 10.18 MB/sec execute 484 sec latency 67.535 ms 2024-04-27T01:16:42.869 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:42 smithi135 podman[54776]: 2024-04-27 01:16:42.627670828 +0000 UTC m=+0.622553442 container remove 9efd15cbc018344babeeb8860c363e1f473823551f36fa1ea4120c8d533b03ff (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:16:42.869 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:42 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-27T01:16:42.869 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:42 smithi135 systemd[1]: Stopped Ceph osd.4 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:16:42.869 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:42 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.4.service: Consumed 4min 44.403s CPU time. 2024-04-27T01:16:43.124 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:42 smithi135 systemd[1]: Starting Ceph osd.4 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:16:43.124 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.022968042 +0000 UTC m=+0.023055443 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:16:43.411 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.137136614 +0000 UTC m=+0.137224004 container create eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:16:43.411 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.423960929 +0000 UTC m=+0.424048330 container init eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:16:43.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 254535 10.18 MB/sec execute 485 sec latency 39.267 ms 2024-04-27T01:16:43.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:43 smithi104 ceph-mon[63925]: pgmap v213: 97 pgs: 7 active+undersized, 5 active+undersized+degraded, 9 stale+active+clean, 76 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.5 MiB/s wr, 371 op/s; 46/1623 objects degraded (2.834%) 2024-04-27T01:16:43.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:43.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:43.718 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:43 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 46/1623 objects degraded (2.834%), 5 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:43.826 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:16:43.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.42834758 +0000 UTC m=+0.428434979 container start eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T01:16:43.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.524007492 +0000 UTC m=+0.524094891 container attach eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T01:16:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-mon[50158]: pgmap v213: 97 pgs: 7 active+undersized, 5 active+undersized+degraded, 9 stale+active+clean, 76 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.5 MiB/s wr, 371 op/s; 46/1623 objects degraded (2.834%) 2024-04-27T01:16:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 46/1623 objects degraded (2.834%), 5 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T01:16:44.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 bash[54872]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate[54887]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-27T01:16:44.364 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:43 smithi135 podman[54872]: 2024-04-27 01:16:43.949849112 +0000 UTC m=+0.949936510 container died eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-27T01:16:44.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 254945 10.19 MB/sec execute 486 sec latency 27.197 ms 2024-04-27T01:16:44.734 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:44 smithi104 ceph-mon[63925]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:44.790 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:44 smithi135 ceph-mon[50158]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:16:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (7m) 32s ago 16m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:16:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (3m) 32s ago 16m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:16:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (3m) 3m ago 15m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:16:44.911 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 32s ago 16m - - 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (10m) 32s ago 10m 41.2M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (10m) 3m ago 10m 16.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (6m) 32s ago 17m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (5m) 3m ago 15m 450M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (5m) 32s ago 17m 51.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (4m) 3m ago 15m 38.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (10m) 32s ago 10m 274M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (10m) 3m ago 10m 83.2M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 32s ago 16m 20.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (6m) 3m ago 15m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (3m) 32s ago 15m 336M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (2m) 32s ago 14m 296M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (90s) 32s ago 14m 230M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:16:44.912 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (43s) 32s ago 13m 15.2M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:16:44.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (13m) 3m ago 13m 562M 2704M 17.2.0 e1d6a67b021e 1c30c1b7d02b 2024-04-27T01:16:44.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (12m) 3m ago 12m 400M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:16:44.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (12m) 3m ago 12m 586M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:16:44.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (12m) 3m ago 12m 496M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:16:44.913 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (5m) 32s ago 15m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:16:45.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 podman[55013]: 2024-04-27 01:16:45.095624869 +0000 UTC m=+1.135458071 container remove eaca68859ca0625d5929fbb48d22753e84b95e047241ab523742b1d204e5dd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5) 2024-04-27T01:16:45.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 podman[55075]: 2024-04-27 01:16:45.212345433 +0000 UTC m=+0.017888237 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:16:45.363 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 podman[55075]: 2024-04-27 01:16:45.330001532 +0000 UTC m=+0.135544302 container create 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-b22e2eb) 2024-04-27T01:16:45.487 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:45 smithi104 ceph-mon[63925]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 593 op/s; 259/1575 objects degraded (16.444%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:45.488 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:45 smithi104 ceph-mon[63925]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:16:45.571 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 4 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:16:45.572 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:16:45.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 255436 10.21 MB/sec execute 487 sec latency 35.394 ms 2024-04-27T01:16:45.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:45 smithi135 ceph-mon[50158]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 593 op/s; 259/1575 objects degraded (16.444%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:45.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:45 smithi135 ceph-mon[50158]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:45.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 podman[55075]: 2024-04-27 01:16:45.557944188 +0000 UTC m=+0.363486959 container init 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T01:16:45.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 podman[55075]: 2024-04-27 01:16:45.560984486 +0000 UTC m=+0.366527268 container start 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T01:16:45.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 ceph-osd[55094]: -- 172.21.15.135:0/975367430 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558228de9680 con 0x558228db2c00 2024-04-27T01:16:45.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 bash[55075]: 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f 2024-04-27T01:16:45.863 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:45 smithi135 systemd[1]: Started Ceph osd.4 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:16:46.178 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "10/23 daemons upgraded", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:16:46.179 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:16:46.362 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:16:46.101+0000 7fcfd07977c0 -1 Falling back to public interface 2024-04-27T01:16:46.514 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:46 smithi104 ceph-mon[63925]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:46.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:46 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2814143661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:46.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:46.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:46.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:46.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 255806 10.22 MB/sec execute 488 sec latency 53.940 ms 2024-04-27T01:16:46.743 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-mon[50158]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:46.743 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2814143661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:46.743 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:46.743 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:46.744 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 259/1575 objects degraded (16.444%), 28 pgs degraded 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout: osd.4 (root=default,host=smithi135) is down 2024-04-27T01:16:46.830 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 259/1575 objects degraded (16.444%), 28 pgs degraded 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.b is active+undersized+degraded, acting [2,5] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-04-27T01:16:46.831 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-04-27T01:16:46.832 INFO:teuthology.orchestra.run.smithi104.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-04-27T01:16:47.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 256279 10.21 MB/sec execute 489 sec latency 60.721 ms 2024-04-27T01:16:47.776 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:47 smithi104 ceph-mon[63925]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.6 MiB/s wr, 437 op/s; 259/1575 objects degraded (16.444%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:47.777 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:47 smithi104 ceph-mon[63925]: from='client.44289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:47.777 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:47 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/662891762' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:16:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:47 smithi135 ceph-mon[50158]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.6 MiB/s wr, 437 op/s; 259/1575 objects degraded (16.444%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:47 smithi135 ceph-mon[50158]: from='client.44289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:16:48.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:47 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/662891762' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:16:48.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 256643 10.20 MB/sec execute 490 sec latency 46.354 ms 2024-04-27T01:16:48.810 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:48 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 263/1641 objects degraded (16.027%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:49.051 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:48 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 263/1641 objects degraded (16.027%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:49.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 256859 10.19 MB/sec execute 491 sec latency 58.361 ms 2024-04-27T01:16:49.834 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:49 smithi104 ceph-mon[63925]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.1 MiB/s wr, 289 op/s; 263/1641 objects degraded (16.027%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:49.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:49.835 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:49 smithi135 ceph-mon[50158]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.1 MiB/s wr, 289 op/s; 263/1641 objects degraded (16.027%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:50.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:50.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 257125 10.19 MB/sec execute 492 sec latency 70.142 ms 2024-04-27T01:16:51.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 257438 10.18 MB/sec execute 493 sec latency 112.192 ms 2024-04-27T01:16:51.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:51 smithi104 ceph-mon[63925]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 506 op/s; 281/1761 objects degraded (15.957%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:51.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:51.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:52.076 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:51 smithi135 ceph-mon[50158]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 506 op/s; 281/1761 objects degraded (15.957%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:52.076 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:52.076 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:52.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 257778 10.17 MB/sec execute 494 sec latency 82.853 ms 2024-04-27T01:16:53.230 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:53 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:16:53.210+0000 7fcfd07977c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-27T01:16:53.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 258045 10.17 MB/sec execute 495 sec latency 53.318 ms 2024-04-27T01:16:53.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:53 smithi104 ceph-mon[63925]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.9 MiB/s wr, 481 op/s; 279/1701 objects degraded (16.402%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:54.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:53 smithi135 ceph-mon[50158]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.9 MiB/s wr, 481 op/s; 279/1701 objects degraded (16.402%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:54.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 258218 10.16 MB/sec execute 496 sec latency 59.563 ms 2024-04-27T01:16:54.862 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:16:54.802+0000 7fcfd07977c0 -1 osd.4 124 log_to_monitors true 2024-04-27T01:16:54.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:54 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 255/1578 objects degraded (16.160%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:54.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:54 smithi135 ceph-mon[50158]: from='osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:16:54.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:54 smithi135 ceph-mon[50158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:16:54.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:54 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 255/1578 objects degraded (16.160%), 28 pgs degraded (PG_DEGRADED) 2024-04-27T01:16:54.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:54 smithi104 ceph-mon[63925]: from='osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:16:54.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:54 smithi104 ceph-mon[63925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:16:55.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 258528 10.16 MB/sec execute 497 sec latency 38.406 ms 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: pgmap v219: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.3 MiB/s wr, 579 op/s; 255/1578 objects degraded (16.160%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:16:55.962 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:16:55.963 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: osdmap e127: 8 total, 7 up, 8 in 2024-04-27T01:16:55.963 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:16:55.963 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:55 smithi104 ceph-mon[63925]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:16:56.112 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:16:55.800+0000 7fcfc1bef700 -1 osd.4 124 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: pgmap v219: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.3 MiB/s wr, 579 op/s; 255/1578 objects degraded (16.160%); 0 B/s, 0 objects/s recovering 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: osdmap e127: 8 total, 7 up, 8 in 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:16:56.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:55 smithi135 ceph-mon[50158]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:16:56.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 258834 10.17 MB/sec execute 498 sec latency 56.019 ms 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: pgmap v220: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 470 op/s; 255/1578 objects degraded (16.160%) 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225] boot 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: osdmap e128: 8 total, 8 up, 8 in 2024-04-27T01:16:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:56 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: pgmap v220: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 470 op/s; 255/1578 objects degraded (16.160%) 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: osd.4 [v2:172.21.15.135:6800/4111259225,v1:172.21.15.135:6801/4111259225] boot 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: osdmap e128: 8 total, 8 up, 8 in 2024-04-27T01:16:57.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:56 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:16:57.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 258960 10.15 MB/sec execute 499 sec latency 811.661 ms 2024-04-27T01:16:58.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:58 smithi104 ceph-mon[63925]: pgmap v223: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.1 MiB/s wr, 328 op/s; 239/1605 objects degraded (14.891%) 2024-04-27T01:16:58.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:58 smithi104 ceph-mon[63925]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T01:16:58.403 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:58.404 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:58.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:58 smithi135 ceph-mon[50158]: pgmap v223: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.1 MiB/s wr, 328 op/s; 239/1605 objects degraded (14.891%) 2024-04-27T01:16:58.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:58 smithi135 ceph-mon[50158]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T01:16:58.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:16:58.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:58.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 259105 10.14 MB/sec execute 500 sec latency 1327.563 ms 2024-04-27T01:16:59.177 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:16:59 smithi104 ceph-mon[63925]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T01:16:59.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:16:59 smithi135 ceph-mon[50158]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T01:16:59.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 259378 10.14 MB/sec execute 501 sec latency 52.590 ms 2024-04-27T01:17:00.189 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:00 smithi104 ceph-mon[63925]: pgmap v226: 97 pgs: 2 active+recovery_wait+degraded, 2 peering, 11 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 7 active+undersized+degraded, 68 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 387 op/s; 176/1659 objects degraded (10.609%); 2.2 MiB/s, 22 objects/s recovering 2024-04-27T01:17:00.189 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:00 smithi104 ceph-mon[63925]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T01:17:00.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:00 smithi135 ceph-mon[50158]: pgmap v226: 97 pgs: 2 active+recovery_wait+degraded, 2 peering, 11 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 7 active+undersized+degraded, 68 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 387 op/s; 176/1659 objects degraded (10.609%); 2.2 MiB/s, 22 objects/s recovering 2024-04-27T01:17:00.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:00 smithi135 ceph-mon[50158]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T01:17:00.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 259600 10.13 MB/sec execute 502 sec latency 35.608 ms 2024-04-27T01:17:01.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:01 smithi104 ceph-mon[63925]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T01:17:01.219 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:01 smithi104 ceph-mon[63925]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T01:17:01.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:01 smithi135 ceph-mon[50158]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T01:17:01.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:01 smithi135 ceph-mon[50158]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T01:17:01.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 259898 10.12 MB/sec execute 503 sec latency 418.968 ms 2024-04-27T01:17:02.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:02 smithi104 ceph-mon[63925]: pgmap v229: 97 pgs: 2 active+recovery_wait+degraded, 2 peering, 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 82 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 359 op/s; 138/1677 objects degraded (8.229%); 2.4 MiB/s, 38 objects/s recovering 2024-04-27T01:17:02.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:02 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 138/1677 objects degraded (8.229%), 10 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:02.284 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:02 smithi104 ceph-mon[63925]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T01:17:02.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:02 smithi135 ceph-mon[50158]: pgmap v229: 97 pgs: 2 active+recovery_wait+degraded, 2 peering, 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 82 active+clean; 338 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 359 op/s; 138/1677 objects degraded (8.229%); 2.4 MiB/s, 38 objects/s recovering 2024-04-27T01:17:02.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:02 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 138/1677 objects degraded (8.229%), 10 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:02.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:02 smithi135 ceph-mon[50158]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T01:17:02.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 260183 10.10 MB/sec execute 504 sec latency 55.001 ms 2024-04-27T01:17:03.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 260435 10.10 MB/sec execute 505 sec latency 359.872 ms 2024-04-27T01:17:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:04 smithi104 ceph-mon[63925]: pgmap v232: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 89 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.5 MiB/s wr, 59 op/s; 117/1686 objects degraded (6.940%); 442 KiB/s, 64 keys/s, 25 objects/s recovering 2024-04-27T01:17:04.333 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:04 smithi104 ceph-mon[63925]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T01:17:04.414 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:04 smithi135 ceph-mon[50158]: pgmap v232: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 89 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.5 MiB/s wr, 59 op/s; 117/1686 objects degraded (6.940%); 442 KiB/s, 64 keys/s, 25 objects/s recovering 2024-04-27T01:17:04.415 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:04 smithi135 ceph-mon[50158]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T01:17:04.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 260728 10.09 MB/sec execute 506 sec latency 878.645 ms 2024-04-27T01:17:05.368 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:05 smithi104 ceph-mon[63925]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T01:17:05.428 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:05 smithi135 ceph-mon[50158]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T01:17:05.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 261113 10.10 MB/sec execute 507 sec latency 60.259 ms 2024-04-27T01:17:06.407 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:06 smithi104 ceph-mon[63925]: pgmap v235: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 507 op/s; 669 KiB/s, 64 keys/s, 42 objects/s recovering 2024-04-27T01:17:06.407 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:06 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 117/1686 objects degraded (6.940%), 7 pgs degraded) 2024-04-27T01:17:06.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:06 smithi135 ceph-mon[50158]: pgmap v235: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 507 op/s; 669 KiB/s, 64 keys/s, 42 objects/s recovering 2024-04-27T01:17:06.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:06 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 117/1686 objects degraded (6.940%), 7 pgs degraded) 2024-04-27T01:17:06.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 261614 10.10 MB/sec execute 508 sec latency 37.840 ms 2024-04-27T01:17:07.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 261975 10.10 MB/sec execute 509 sec latency 41.606 ms 2024-04-27T01:17:08.187 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:08 smithi104 ceph-mon[63925]: pgmap v236: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 517 op/s; 459 KiB/s, 44 keys/s, 29 objects/s recovering 2024-04-27T01:17:08.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:08 smithi135 ceph-mon[50158]: pgmap v236: 97 pgs: 1 peering, 96 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 517 op/s; 459 KiB/s, 44 keys/s, 29 objects/s recovering 2024-04-27T01:17:08.685 INFO:teuthology.orchestra.run.smithi104.stdout: 5 262386 10.12 MB/sec execute 510 sec latency 29.121 ms 2024-04-27T01:17:09.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 262723 10.12 MB/sec execute 511 sec latency 60.015 ms 2024-04-27T01:17:10.218 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:10 smithi104 ceph-mon[63925]: pgmap v237: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 579 op/s; 268 KiB/s, 19 objects/s recovering 2024-04-27T01:17:10.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:10 smithi135 ceph-mon[50158]: pgmap v237: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 579 op/s; 268 KiB/s, 19 objects/s recovering 2024-04-27T01:17:10.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 262983 10.11 MB/sec execute 512 sec latency 49.661 ms 2024-04-27T01:17:11.227 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:17:11.227 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:11 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:17:11.228 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:11 smithi104 ceph-mon[63925]: Upgrade: osd.5 is safe to restart 2024-04-27T01:17:11.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:17:11.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:11 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-27T01:17:11.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:11 smithi135 ceph-mon[50158]: Upgrade: osd.5 is safe to restart 2024-04-27T01:17:11.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 263212 10.10 MB/sec execute 513 sec latency 47.464 ms 2024-04-27T01:17:12.241 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:12 smithi104 ceph-mon[63925]: pgmap v238: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 507 op/s; 230 KiB/s, 16 objects/s recovering 2024-04-27T01:17:12.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:12 smithi135 ceph-mon[50158]: pgmap v238: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.9 MiB/s wr, 507 op/s; 230 KiB/s, 16 objects/s recovering 2024-04-27T01:17:12.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 263509 10.09 MB/sec execute 514 sec latency 68.273 ms 2024-04-27T01:17:13.514 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: Upgrade: Updating osd.5 2024-04-27T01:17:13.514 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:13.514 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:17:13.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:13.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: Deploying daemon osd.5 on smithi135 2024-04-27T01:17:13.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:13.515 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:13.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 263822 10.08 MB/sec execute 515 sec latency 58.015 ms 2024-04-27T01:17:13.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: Upgrade: Updating osd.5 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: Deploying daemon osd.5 on smithi135 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:13.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:14.531 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:14 smithi104 ceph-mon[63925]: pgmap v239: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.3 MiB/s wr, 588 op/s; 186 KiB/s, 13 objects/s recovering 2024-04-27T01:17:14.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 264100 10.07 MB/sec execute 516 sec latency 91.967 ms 2024-04-27T01:17:14.701 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:14 smithi135 ceph-mon[50158]: pgmap v239: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.3 MiB/s wr, 588 op/s; 186 KiB/s, 13 objects/s recovering 2024-04-27T01:17:15.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 264444 10.08 MB/sec execute 517 sec latency 52.051 ms 2024-04-27T01:17:16.363 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:16 smithi135 systemd[1]: Stopping Ceph osd.5 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:17:16.569 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:16 smithi104 ceph-mon[63925]: pgmap v240: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 457 op/s 2024-04-27T01:17:16.569 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:16 smithi104 ceph-mon[63925]: osd.5 marked itself down and dead 2024-04-27T01:17:16.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 264804 10.07 MB/sec execute 518 sec latency 161.408 ms 2024-04-27T01:17:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:16 smithi135 ceph-mon[50158]: pgmap v240: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 457 op/s 2024-04-27T01:17:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:16 smithi135 ceph-mon[50158]: osd.5 marked itself down and dead 2024-04-27T01:17:16.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[31299]: 2024-04-27T01:17:16.432+0000 7fcae3413700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:17:16.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[31299]: 2024-04-27T01:17:16.432+0000 7fcae3413700 -1 osd.5 136 *** Got signal Terminated *** 2024-04-27T01:17:16.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[31299]: 2024-04-27T01:17:16.432+0000 7fcae3413700 -1 osd.5 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:17:17.190 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:17:17.486 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:17 smithi135 podman[56891]: 2024-04-27 01:17:17.166479502 +0000 UTC m=+0.885466332 container died 7f55c6f902ea514c54ba7bec3d193b207d2fd0e238ec4709f299c3ab87de4c58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, io.openshift.expose-services=, release=754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD) 2024-04-27T01:17:17.486 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:17 smithi135 podman[56891]: 2024-04-27 01:17:17.499300359 +0000 UTC m=+1.218287206 container cleanup 7f55c6f902ea514c54ba7bec3d193b207d2fd0e238ec4709f299c3ab87de4c58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, version=8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-27T01:17:17.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 264866 10.06 MB/sec execute 519 sec latency 1030.627 ms 2024-04-27T01:17:17.794 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:17 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:17:17.795 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:17 smithi104 ceph-mon[63925]: osdmap e137: 8 total, 7 up, 8 in 2024-04-27T01:17:17.795 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:17 smithi104 ceph-mon[63925]: osdmap e138: 8 total, 7 up, 8 in 2024-04-27T01:17:17.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:17 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:17:17.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:17 smithi135 ceph-mon[50158]: osdmap e137: 8 total, 7 up, 8 in 2024-04-27T01:17:17.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:17 smithi135 ceph-mon[50158]: osdmap e138: 8 total, 7 up, 8 in 2024-04-27T01:17:17.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:17 smithi135 bash[56891]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5 2024-04-27T01:17:17.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:17 smithi135 podman[56908]: 2024-04-27 01:17:17.717428005 +0000 UTC m=+0.547291844 container remove 7f55c6f902ea514c54ba7bec3d193b207d2fd0e238ec4709f299c3ab87de4c58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8) 2024-04-27T01:17:18.124 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:17:18.124 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (8m) 65s ago 17m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (4m) 65s ago 17m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (4m) 28s ago 16m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 65s ago 16m - - 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (11m) 65s ago 11m 41.2M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (11m) 28s ago 11m 17.5M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (7m) 65s ago 18m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (6m) 28s ago 16m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (5m) 65s ago 18m 51.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (5m) 28s ago 16m 42.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (10m) 65s ago 10m 274M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (10m) 28s ago 10m 85.7M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 65s ago 16m 20.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 28s ago 16m 17.8M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:17:18.125 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (3m) 65s ago 15m 336M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (2m) 65s ago 15m 296M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (2m) 65s ago 14m 230M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (76s) 65s ago 14m 15.2M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (32s) 28s ago 13m 14.9M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (13m) 28s ago 13m 600M 2704M 17.2.0 e1d6a67b021e 7f55c6f902ea 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (13m) 28s ago 13m 538M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (12m) 28s ago 12m 796M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:17:18.126 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (6m) 65s ago 15m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:17:18.484 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.070202884 +0000 UTC m=+0.021486879 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:18.484 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.185155139 +0000 UTC m=+0.136439113 container create a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:17:18.484 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.396469064 +0000 UTC m=+0.347753038 container init a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb) 2024-04-27T01:17:18.484 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.399918585 +0000 UTC m=+0.351202559 container start a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:17:18.485 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.496744488 +0000 UTC m=+0.448028467 container attach a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:17:18.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 265184 10.05 MB/sec execute 520 sec latency 55.143 ms 2024-04-27T01:17:18.808 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:17:18.808 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:17:18.808 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:17:18.808 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:18.808 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 5 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 9 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:17:18.809 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:17:18.834 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 conmon[57014]: conmon a9993b35d49efa349c40 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55.scope/container/memory.events 2024-04-27T01:17:18.834 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:18 smithi135 podman[56999]: 2024-04-27 01:17:18.736831491 +0000 UTC m=+0.688115466 container died a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb) 2024-04-27T01:17:19.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:18 smithi104 ceph-mon[63925]: pgmap v242: 97 pgs: 10 peering, 9 stale+active+clean, 78 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 488 op/s 2024-04-27T01:17:19.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:18 smithi104 ceph-mon[63925]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:19.060 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:18 smithi104 ceph-mon[63925]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:19.060 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:18 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:19.060 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:18 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2261152662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:19.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:18 smithi135 ceph-mon[50158]: pgmap v242: 97 pgs: 10 peering, 9 stale+active+clean, 78 active+clean; 329 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 488 op/s 2024-04-27T01:17:19.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:18 smithi135 ceph-mon[50158]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:19.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:18 smithi135 ceph-mon[50158]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:19.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:18 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:19.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:18 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2261152662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:17:19.313 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:17:19.314 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "11/23 daemons upgraded", 2024-04-27T01:17:19.314 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:17:19.314 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:17:19.314 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:17:19.541 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 podman[57025]: 2024-04-27 01:17:19.326479243 +0000 UTC m=+0.580641015 container remove a9993b35d49efa349c40a3fee2206467aaa6393dceb3082e17fda1a6b05dbd55 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T01:17:19.541 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-27T01:17:19.541 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 systemd[1]: Stopped Ceph osd.5 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:17:19.541 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.5.service: Consumed 5min 32.181s CPU time. 2024-04-27T01:17:19.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 265436 10.05 MB/sec execute 521 sec latency 41.478 ms 2024-04-27T01:17:19.841 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 systemd[1]: Starting Ceph osd.5 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:17:19.841 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 podman[57128]: 2024-04-27 01:17:19.684734595 +0000 UTC m=+0.019621765 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:19.841 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:19 smithi135 podman[57128]: 2024-04-27 01:17:19.802663866 +0000 UTC m=+0.137551025 container create 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:17:19.926 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 83/1569 objects degraded (5.290%), 12 pgs degraded 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: osd.5 (root=default,host=smithi135) is down 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 83/1569 objects degraded (5.290%), 12 pgs degraded 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.b is active+undersized+degraded, acting [2,4] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-04-27T01:17:19.927 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-04-27T01:17:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-04-27T01:17:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-04-27T01:17:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-04-27T01:17:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-04-27T01:17:19.928 INFO:teuthology.orchestra.run.smithi104.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-04-27T01:17:20.084 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:19 smithi104 ceph-mon[63925]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:20.104 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:19 smithi135 ceph-mon[50158]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:20.104 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 podman[57128]: 2024-04-27 01:17:20.016072759 +0000 UTC m=+0.350959920 container init 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:17:20.104 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 podman[57128]: 2024-04-27 01:17:20.020329732 +0000 UTC m=+0.355216890 container start 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:17:20.104 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 podman[57128]: 2024-04-27 01:17:20.117311107 +0000 UTC m=+0.452198265 container attach 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:17:20.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 265794 10.06 MB/sec execute 522 sec latency 36.031 ms 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T01:17:20.843 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 bash[57128]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate[57143]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-27T01:17:20.844 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:20 smithi135 podman[57128]: 2024-04-27 01:17:20.552851367 +0000 UTC m=+0.887738536 container died 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:17:20.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:20 smithi104 ceph-mon[63925]: pgmap v244: 97 pgs: 6 active+undersized, 12 active+undersized+degraded, 10 peering, 3 stale+active+clean, 66 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 561 op/s; 83/1569 objects degraded (5.290%) 2024-04-27T01:17:20.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:20 smithi104 ceph-mon[63925]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:20.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:20 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 83/1569 objects degraded (5.290%), 12 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:20.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:20 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2066247228' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:17:21.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-mon[50158]: pgmap v244: 97 pgs: 6 active+undersized, 12 active+undersized+degraded, 10 peering, 3 stale+active+clean, 66 active+clean; 328 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.8 MiB/s wr, 561 op/s; 83/1569 objects degraded (5.290%) 2024-04-27T01:17:21.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-mon[50158]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:21.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 83/1569 objects degraded (5.290%), 12 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:21.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:20 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2066247228' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:17:21.506 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 podman[57265]: 2024-04-27 01:17:21.162640098 +0000 UTC m=+0.601547826 container remove 013b50db45880e5bd4ec31c6dfa3511c453e65d63f54695e56683653917f2c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-activate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T01:17:21.507 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 podman[57327]: 2024-04-27 01:17:21.29625571 +0000 UTC m=+0.019456017 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:21.507 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 podman[57327]: 2024-04-27 01:17:21.405279287 +0000 UTC m=+0.128479593 container create bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:17:21.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 266286 10.07 MB/sec execute 523 sec latency 29.062 ms 2024-04-27T01:17:21.862 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 podman[57327]: 2024-04-27 01:17:21.633473198 +0000 UTC m=+0.356673524 container init bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-27T01:17:21.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 podman[57327]: 2024-04-27 01:17:21.636984034 +0000 UTC m=+0.360184348 container start bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-27T01:17:21.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 ceph-osd[57346]: -- 172.21.15.135:0/3607775632 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f1f51f680 con 0x557f1f4e9000 2024-04-27T01:17:21.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 bash[57327]: bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc 2024-04-27T01:17:21.863 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:21 smithi135 systemd[1]: Started Ceph osd.5 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:17:22.612 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:22 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:17:22.179+0000 7fbf0d7c97c0 -1 Falling back to public interface 2024-04-27T01:17:22.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 266585 10.07 MB/sec execute 524 sec latency 47.909 ms 2024-04-27T01:17:22.880 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:22 smithi104 ceph-mon[63925]: pgmap v245: 97 pgs: 7 active+undersized, 15 active+undersized+degraded, 10 peering, 65 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.5 MiB/s wr, 359 op/s; 108/1584 objects degraded (6.818%) 2024-04-27T01:17:22.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:22.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:22.881 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:22 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:23.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:22 smithi135 ceph-mon[50158]: pgmap v245: 97 pgs: 7 active+undersized, 15 active+undersized+degraded, 10 peering, 65 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.5 MiB/s wr, 359 op/s; 108/1584 objects degraded (6.818%) 2024-04-27T01:17:23.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:23.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:23.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:22 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:23.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 266876 10.06 MB/sec execute 525 sec latency 37.348 ms 2024-04-27T01:17:24.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 267252 10.05 MB/sec execute 526 sec latency 76.380 ms 2024-04-27T01:17:24.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:24 smithi135 ceph-mon[50158]: pgmap v246: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 359 op/s; 172/1605 objects degraded (10.717%) 2024-04-27T01:17:24.910 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:24 smithi104 ceph-mon[63925]: pgmap v246: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 359 op/s; 172/1605 objects degraded (10.717%) 2024-04-27T01:17:25.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 267562 10.04 MB/sec execute 527 sec latency 47.407 ms 2024-04-27T01:17:25.930 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:25 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 185/1719 objects degraded (10.762%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:26.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:25 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 185/1719 objects degraded (10.762%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:26.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 267862 10.04 MB/sec execute 528 sec latency 60.342 ms 2024-04-27T01:17:26.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:26 smithi104 ceph-mon[63925]: pgmap v247: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.2 MiB/s wr, 521 op/s; 185/1719 objects degraded (10.762%) 2024-04-27T01:17:26.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:26 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:26.940 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:26 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:27.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:26 smithi135 ceph-mon[50158]: pgmap v247: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.2 MiB/s wr, 521 op/s; 185/1719 objects degraded (10.762%) 2024-04-27T01:17:27.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:26 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:27.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:26 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:27.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 268216 10.04 MB/sec execute 529 sec latency 76.206 ms 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: pgmap v248: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.8 MiB/s wr, 444 op/s; 186/1677 objects degraded (11.091%) 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:28.451 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: pgmap v248: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.8 MiB/s wr, 444 op/s; 186/1677 objects degraded (11.091%) 2024-04-27T01:17:28.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:28.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:28.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 268663 10.04 MB/sec execute 530 sec latency 46.027 ms 2024-04-27T01:17:29.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 269003 10.04 MB/sec execute 531 sec latency 43.883 ms 2024-04-27T01:17:30.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:30 smithi104 ceph-mon[63925]: pgmap v249: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 492 op/s; 187/1623 objects degraded (11.522%) 2024-04-27T01:17:30.659 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:30 smithi135 ceph-mon[50158]: pgmap v249: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 492 op/s; 187/1623 objects degraded (11.522%) 2024-04-27T01:17:30.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 269367 10.05 MB/sec execute 532 sec latency 58.092 ms 2024-04-27T01:17:31.057 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:30 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:17:30.670+0000 7fbf0d7c97c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-27T01:17:31.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 269820 10.06 MB/sec execute 533 sec latency 32.824 ms 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: pgmap v250: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 411 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: pgmap v251: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 476 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: pgmap v252: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.7 MiB/s wr, 469 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.138 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 185/1599 objects degraded (11.570%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:17:32.139 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:32 smithi104 ceph-mon[63925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:17:32.362 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:17:32.095+0000 7fbf0d7c97c0 -1 osd.5 136 log_to_monitors true 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: pgmap v250: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 411 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: pgmap v251: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 476 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: pgmap v252: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 327 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.7 MiB/s wr, 469 op/s; 185/1599 objects degraded (11.570%) 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 185/1599 objects degraded (11.570%), 23 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:17:32.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-mon[50158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:17:32.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 270193 10.06 MB/sec execute 534 sec latency 39.829 ms 2024-04-27T01:17:32.862 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:17:32 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:17:32.424+0000 7fbefec21700 -1 osd.5 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:17:33.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:33 smithi104 ceph-mon[63925]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:17:33.397 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:33 smithi104 ceph-mon[63925]: osdmap e139: 8 total, 7 up, 8 in 2024-04-27T01:17:33.398 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:33 smithi104 ceph-mon[63925]: from='osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:17:33.398 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:33 smithi104 ceph-mon[63925]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:17:33.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 270677 10.05 MB/sec execute 535 sec latency 262.521 ms 2024-04-27T01:17:33.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:33 smithi135 ceph-mon[50158]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:17:33.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:33 smithi135 ceph-mon[50158]: osdmap e139: 8 total, 7 up, 8 in 2024-04-27T01:17:33.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:33 smithi135 ceph-mon[50158]: from='osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:17:33.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:33 smithi135 ceph-mon[50158]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: pgmap v254: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 194/1668 objects degraded (11.631%) 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078] boot 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:17:34.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:34 smithi104 ceph-mon[63925]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T01:17:34.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 270714 10.03 MB/sec execute 536 sec latency 1262.547 ms 2024-04-27T01:17:34.763 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: pgmap v254: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 65 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 433 op/s; 194/1668 objects degraded (11.631%) 2024-04-27T01:17:34.763 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:17:34.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: osd.5 [v2:172.21.15.135:6808/3345637078,v1:172.21.15.135:6809/3345637078] boot 2024-04-27T01:17:34.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T01:17:34.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:17:34.764 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:34 smithi135 ceph-mon[50158]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T01:17:35.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 270741 10.02 MB/sec execute 537 sec latency 2010.257 ms 2024-04-27T01:17:36.460 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:36 smithi104 ceph-mon[63925]: pgmap v257: 97 pgs: 12 peering, 6 active+undersized, 14 active+undersized+degraded, 65 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 7.4 MiB/s wr, 661 op/s; 119/1773 objects degraded (6.712%) 2024-04-27T01:17:36.460 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:36 smithi104 ceph-mon[63925]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T01:17:36.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 271015 10.01 MB/sec execute 538 sec latency 275.552 ms 2024-04-27T01:17:36.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:36 smithi135 ceph-mon[50158]: pgmap v257: 97 pgs: 12 peering, 6 active+undersized, 14 active+undersized+degraded, 65 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 7.4 MiB/s wr, 661 op/s; 119/1773 objects degraded (6.712%) 2024-04-27T01:17:36.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:36 smithi135 ceph-mon[50158]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T01:17:37.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:37 smithi104 ceph-mon[63925]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T01:17:37.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:37 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 119/1773 objects degraded (6.712%), 14 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:37.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:37 smithi104 ceph-mon[63925]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T01:17:37.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 271306 10.01 MB/sec execute 539 sec latency 778.651 ms 2024-04-27T01:17:37.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:37 smithi135 ceph-mon[50158]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T01:17:37.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:37 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 119/1773 objects degraded (6.712%), 14 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:37.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:37 smithi135 ceph-mon[50158]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T01:17:38.439 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:38 smithi104 ceph-mon[63925]: pgmap v260: 97 pgs: 13 peering, 3 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 71 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 6.1 MiB/s wr, 676 op/s; 104/1653 objects degraded (6.292%); 485 KiB/s, 8 objects/s recovering 2024-04-27T01:17:38.439 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:38 smithi104 ceph-mon[63925]: osdmap e145: 8 total, 8 up, 8 in 2024-04-27T01:17:38.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 271494 10.00 MB/sec execute 540 sec latency 1013.940 ms 2024-04-27T01:17:38.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:38 smithi135 ceph-mon[50158]: pgmap v260: 97 pgs: 13 peering, 3 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+undersized+degraded, 71 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 6.1 MiB/s wr, 676 op/s; 104/1653 objects degraded (6.292%); 485 KiB/s, 8 objects/s recovering 2024-04-27T01:17:38.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:38 smithi135 ceph-mon[50158]: osdmap e145: 8 total, 8 up, 8 in 2024-04-27T01:17:39.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 271906 10.00 MB/sec execute 541 sec latency 43.835 ms 2024-04-27T01:17:39.706 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:39 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/1653 objects degraded (6.292%), 10 pgs degraded) 2024-04-27T01:17:39.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:39 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 104/1653 objects degraded (6.292%), 10 pgs degraded) 2024-04-27T01:17:40.479 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:40 smithi104 ceph-mon[63925]: pgmap v263: 97 pgs: 4 peering, 93 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.9 MiB/s wr, 442 op/s; 3.9 MiB/s, 46 objects/s recovering 2024-04-27T01:17:40.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 272373 10.00 MB/sec execute 542 sec latency 34.555 ms 2024-04-27T01:17:40.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:40 smithi135 ceph-mon[50158]: pgmap v263: 97 pgs: 4 peering, 93 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 3.9 MiB/s wr, 442 op/s; 3.9 MiB/s, 46 objects/s recovering 2024-04-27T01:17:41.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 272738 10.01 MB/sec execute 543 sec latency 33.665 ms 2024-04-27T01:17:42.522 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:42 smithi104 ceph-mon[63925]: pgmap v264: 97 pgs: 4 peering, 93 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.0 MiB/s wr, 342 op/s; 2.7 MiB/s, 32 objects/s recovering 2024-04-27T01:17:42.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 273132 10.02 MB/sec execute 544 sec latency 34.446 ms 2024-04-27T01:17:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:42 smithi135 ceph-mon[50158]: pgmap v264: 97 pgs: 4 peering, 93 active+clean; 330 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.0 MiB/s wr, 342 op/s; 2.7 MiB/s, 32 objects/s recovering 2024-04-27T01:17:43.538 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:43.538 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:43 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:43.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 273489 10.02 MB/sec execute 545 sec latency 57.088 ms 2024-04-27T01:17:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:43.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:43 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:44.572 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:44 smithi104 ceph-mon[63925]: pgmap v265: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 451 op/s; 2.4 MiB/s, 33 objects/s recovering 2024-04-27T01:17:44.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 273761 10.02 MB/sec execute 546 sec latency 62.272 ms 2024-04-27T01:17:44.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:44 smithi135 ceph-mon[50158]: pgmap v265: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 451 op/s; 2.4 MiB/s, 33 objects/s recovering 2024-04-27T01:17:45.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 274048 10.00 MB/sec execute 547 sec latency 47.147 ms 2024-04-27T01:17:46.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:46 smithi135 ceph-mon[50158]: pgmap v266: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 372 op/s; 1.8 MiB/s, 23 objects/s recovering 2024-04-27T01:17:46.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:46.635 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:46 smithi104 ceph-mon[63925]: pgmap v266: 97 pgs: 97 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 372 op/s; 1.8 MiB/s, 23 objects/s recovering 2024-04-27T01:17:46.635 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:46.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 274325 9.99 MB/sec execute 548 sec latency 51.925 ms 2024-04-27T01:17:47.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:47 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:47.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:47 smithi135 ceph-mon[50158]: Upgrade: osd.6 is safe to restart 2024-04-27T01:17:47.653 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:47 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-27T01:17:47.653 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:47 smithi104 ceph-mon[63925]: Upgrade: osd.6 is safe to restart 2024-04-27T01:17:47.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 274569 9.98 MB/sec execute 549 sec latency 42.902 ms 2024-04-27T01:17:48.665 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:48 smithi104 ceph-mon[63925]: pgmap v267: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 495 op/s; 1.4 MiB/s, 19 objects/s recovering 2024-04-27T01:17:48.665 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:48.665 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:17:48.665 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:48 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:48.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 274905 9.98 MB/sec execute 550 sec latency 84.493 ms 2024-04-27T01:17:48.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:48 smithi135 ceph-mon[50158]: pgmap v267: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 495 op/s; 1.4 MiB/s, 19 objects/s recovering 2024-04-27T01:17:48.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:48.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:17:48.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:48 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:49.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:49 smithi104 ceph-mon[63925]: Upgrade: Updating osd.6 2024-04-27T01:17:49.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:49 smithi104 ceph-mon[63925]: Deploying daemon osd.6 on smithi135 2024-04-27T01:17:49.686 INFO:teuthology.orchestra.run.smithi104.stdout: 5 275301 9.98 MB/sec execute 551 sec latency 36.418 ms 2024-04-27T01:17:49.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:49 smithi135 ceph-mon[50158]: Upgrade: Updating osd.6 2024-04-27T01:17:49.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:49 smithi135 ceph-mon[50158]: Deploying daemon osd.6 on smithi135 2024-04-27T01:17:50.419 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:17:50.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:50 smithi135 ceph-mon[50158]: pgmap v268: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.9 MiB/s wr, 482 op/s; 1.2 MiB/s, 11 objects/s recovering 2024-04-27T01:17:50.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 275648 9.98 MB/sec execute 552 sec latency 47.766 ms 2024-04-27T01:17:50.719 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:50 smithi104 ceph-mon[63925]: pgmap v268: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.9 MiB/s wr, 482 op/s; 1.2 MiB/s, 11 objects/s recovering 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (8m) 99s ago 17m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (4m) 99s ago 17m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (4m) 25s ago 17m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 99s ago 17m - - 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (11m) 99s ago 11m 41.2M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (11m) 25s ago 11m 17.6M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (7m) 99s ago 18m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (6m) 25s ago 16m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (6m) 99s ago 18m 51.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:17:51.472 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (5m) 25s ago 16m 44.1M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (11m) 99s ago 11m 274M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (11m) 25s ago 11m 85.8M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 99s ago 17m 20.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (7m) 25s ago 16m 22.0M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (4m) 99s ago 16m 336M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (3m) 99s ago 15m 296M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (2m) 99s ago 15m 230M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (109s) 99s ago 14m 15.2M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (66s) 25s ago 14m 259M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (30s) 25s ago 14m 11.6M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (13m) 25s ago 13m 569M 2704M 17.2.0 e1d6a67b021e 890fdc18f38e 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (13m) 25s ago 13m 587M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:17:51.473 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (6m) 99s ago 16m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:17:51.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 275963 9.98 MB/sec execute 553 sec latency 47.299 ms 2024-04-27T01:17:51.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:51 smithi104 ceph-mon[63925]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:51.732 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:51 smithi104 ceph-mon[63925]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:51.789 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:51 smithi135 ceph-mon[50158]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:51.789 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:51 smithi135 ceph-mon[50158]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:17:51.992 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 6 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 10 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:17:51.993 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:17:52.078 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:51 smithi135 systemd[1]: Stopping Ceph osd.6 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:17:52.362 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[34286]: 2024-04-27T01:17:52.113+0000 7fb5adb35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:17:52.362 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[34286]: 2024-04-27T01:17:52.113+0000 7fb5adb35700 -1 osd.6 145 *** Got signal Terminated *** 2024-04-27T01:17:52.363 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[34286]: 2024-04-27T01:17:52.113+0000 7fb5adb35700 -1 osd.6 145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "12/23 daemons upgraded", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:17:52.450 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:17:52.451 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:17:52.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 276177 9.97 MB/sec execute 554 sec latency 173.801 ms 2024-04-27T01:17:52.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-mon[50158]: pgmap v269: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 422 op/s; 59 KiB/s, 3 objects/s recovering 2024-04-27T01:17:52.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-mon[50158]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:52.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/369225701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:52.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:52 smithi135 ceph-mon[50158]: osd.6 marked itself down and dead 2024-04-27T01:17:52.874 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down 2024-04-27T01:17:52.874 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:17:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:17:52.874 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-27T01:17:52.874 INFO:teuthology.orchestra.run.smithi104.stdout: osd.6 (root=default,host=smithi135) is down 2024-04-27T01:17:52.874 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:52 smithi104 ceph-mon[63925]: pgmap v269: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.2 MiB/s wr, 422 op/s; 59 KiB/s, 3 objects/s recovering 2024-04-27T01:17:52.874 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:52 smithi104 ceph-mon[63925]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:52.874 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:52 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/369225701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:17:52.874 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:52 smithi104 ceph-mon[63925]: osd.6 marked itself down and dead 2024-04-27T01:17:53.362 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:53 smithi135 podman[59120]: 2024-04-27 01:17:53.193170691 +0000 UTC m=+1.200291561 container died 890fdc18f38efc68585fba214621a8b221c7ae49043642e09e94807754e749a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.expose-services=, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, name=centos-stream, io.buildah.version=1.19.8) 2024-04-27T01:17:53.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 276211 9.96 MB/sec execute 555 sec latency 1078.102 ms 2024-04-27T01:17:53.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:53.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:17:53.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: osdmap e146: 8 total, 7 up, 8 in 2024-04-27T01:17:53.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:53.787 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1640264320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:17:53.788 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:53 smithi104 ceph-mon[63925]: osdmap e147: 8 total, 7 up, 8 in 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: from='client.44329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: osdmap e146: 8 total, 7 up, 8 in 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1640264320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:17:53.851 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:53 smithi135 ceph-mon[50158]: osdmap e147: 8 total, 7 up, 8 in 2024-04-27T01:17:53.852 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:53 smithi135 podman[59120]: 2024-04-27 01:17:53.554559307 +0000 UTC m=+1.561680184 container cleanup 890fdc18f38efc68585fba214621a8b221c7ae49043642e09e94807754e749a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870) 2024-04-27T01:17:53.852 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:53 smithi135 bash[59120]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6 2024-04-27T01:17:54.112 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:53 smithi135 podman[59146]: 2024-04-27 01:17:53.864124226 +0000 UTC m=+0.662110513 container remove 890fdc18f38efc68585fba214621a8b221c7ae49043642e09e94807754e749a8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, version=8, name=centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, architecture=x86_64) 2024-04-27T01:17:54.489 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.127966834 +0000 UTC m=+0.020817949 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:54.489 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.256913887 +0000 UTC m=+0.149764964 container create 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-27T01:17:54.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 276609 9.97 MB/sec execute 556 sec latency 55.590 ms 2024-04-27T01:17:54.834 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.502533258 +0000 UTC m=+0.395384331 container init 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-27T01:17:54.835 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.507417555 +0000 UTC m=+0.400268626 container start 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-27T01:17:54.835 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.601426191 +0000 UTC m=+0.494277275 container attach 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:17:55.066 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:54 smithi104 ceph-mon[63925]: pgmap v271: 97 pgs: 10 stale+active+clean, 87 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-04-27T01:17:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:54 smithi135 ceph-mon[50158]: pgmap v271: 97 pgs: 10 stale+active+clean, 87 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 444 op/s 2024-04-27T01:17:55.112 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:54 smithi135 podman[59235]: 2024-04-27 01:17:54.847605223 +0000 UTC m=+0.740456299 container died 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-27T01:17:55.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 276998 9.97 MB/sec execute 557 sec latency 39.754 ms 2024-04-27T01:17:55.862 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:55 smithi135 podman[59261]: 2024-04-27 01:17:55.539744991 +0000 UTC m=+0.686388136 container remove 116c2974c74cc10431b1c7fa7a800645fc8e2076b06027a3791ab208c0b85b47 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb) 2024-04-27T01:17:55.862 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:55 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-27T01:17:55.863 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:55 smithi135 systemd[1]: Stopped Ceph osd.6 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:17:55.863 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:55 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.6.service: Consumed 5min 29.585s CPU time. 2024-04-27T01:17:55.863 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:55 smithi135 systemd[1]: Starting Ceph osd.6 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:17:56.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:55 smithi104 ceph-mon[63925]: pgmap v273: 97 pgs: 7 active+undersized+degraded, 7 stale+active+clean, 83 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 365 op/s; 63/1626 objects degraded (3.875%) 2024-04-27T01:17:56.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:55 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 63/1626 objects degraded (3.875%), 7 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:56.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:55 smithi135 ceph-mon[50158]: pgmap v273: 97 pgs: 7 active+undersized+degraded, 7 stale+active+clean, 83 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 365 op/s; 63/1626 objects degraded (3.875%) 2024-04-27T01:17:56.272 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:55 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 63/1626 objects degraded (3.875%), 7 pgs degraded (PG_DEGRADED) 2024-04-27T01:17:56.273 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:55.907706167 +0000 UTC m=+0.019203039 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:56.273 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:56.03237823 +0000 UTC m=+0.143875099 container create 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-27T01:17:56.612 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:56.286073938 +0000 UTC m=+0.397570817 container init 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T01:17:56.612 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:56.289842176 +0000 UTC m=+0.401339052 container start 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T01:17:56.613 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:56.402429835 +0000 UTC m=+0.513926711 container attach 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-27T01:17:56.689 INFO:teuthology.orchestra.run.smithi104.stdout: 5 277317 9.97 MB/sec execute 558 sec latency 64.225 ms 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T01:17:57.113 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 bash[59394]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate[59409]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-27T01:17:57.114 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:56 smithi135 podman[59394]: 2024-04-27 01:17:56.826941251 +0000 UTC m=+0.938438114 container died 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True) 2024-04-27T01:17:57.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 277621 9.96 MB/sec execute 559 sec latency 49.368 ms 2024-04-27T01:17:57.928 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:57 smithi135 podman[59526]: 2024-04-27 01:17:57.690847058 +0000 UTC m=+0.858489043 container remove 77d3faf8a25fe54ec58969668575ab1c42fb61d1f9e9845054d818b1416a4256 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T01:17:57.928 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:57 smithi135 podman[59587]: 2024-04-27 01:17:57.823366002 +0000 UTC m=+0.021417527 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:17:57.928 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:57 smithi135 podman[59587]: 2024-04-27 01:17:57.941674497 +0000 UTC m=+0.139726058 container create 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb) 2024-04-27T01:17:58.199 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 podman[59587]: 2024-04-27 01:17:58.186267465 +0000 UTC m=+0.384318975 container init 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-27T01:17:58.199 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 podman[59587]: 2024-04-27 01:17:58.190559281 +0000 UTC m=+0.388610794 container start 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T01:17:58.480 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-osd[59605]: -- 172.21.15.135:0/3728491353 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5644ec507680 con 0x5644ec4d0c00 2024-04-27T01:17:58.480 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 bash[59587]: 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 2024-04-27T01:17:58.481 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 systemd[1]: Started Ceph osd.6 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:17:58.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: pgmap v274: 97 pgs: 8 active+undersized, 20 active+undersized+degraded, 2 stale+active+clean, 67 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s wr, 352 op/s; 176/1683 objects degraded (10.458%) 2024-04-27T01:17:58.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:58.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.627 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.628 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:17:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:58.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 278005 9.96 MB/sec execute 560 sec latency 65.388 ms 2024-04-27T01:17:58.862 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:17:58.727+0000 7f04d93437c0 -1 Falling back to public interface 2024-04-27T01:17:58.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: pgmap v274: 97 pgs: 8 active+undersized, 20 active+undersized+degraded, 2 stale+active+clean, 67 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s wr, 352 op/s; 176/1683 objects degraded (10.458%) 2024-04-27T01:17:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:17:58.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:17:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:59.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 278337 9.95 MB/sec execute 561 sec latency 41.596 ms 2024-04-27T01:18:00.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 278770 9.96 MB/sec execute 562 sec latency 73.444 ms 2024-04-27T01:18:00.690 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:00 smithi104 ceph-mon[63925]: pgmap v275: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.9 MiB/s wr, 574 op/s; 209/1707 objects degraded (12.244%) 2024-04-27T01:18:00.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:00 smithi135 ceph-mon[50158]: pgmap v275: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.9 MiB/s wr, 574 op/s; 209/1707 objects degraded (12.244%) 2024-04-27T01:18:01.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 279152 9.95 MB/sec execute 563 sec latency 58.095 ms 2024-04-27T01:18:02.489 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:02 smithi104 ceph-mon[63925]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.7 MiB/s wr, 501 op/s; 201/1659 objects degraded (12.116%) 2024-04-27T01:18:02.489 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:02 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 201/1659 objects degraded (12.116%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:02.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 279500 9.95 MB/sec execute 564 sec latency 43.812 ms 2024-04-27T01:18:02.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:02 smithi135 ceph-mon[50158]: pgmap v276: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.7 MiB/s wr, 501 op/s; 201/1659 objects degraded (12.116%) 2024-04-27T01:18:02.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:02 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 201/1659 objects degraded (12.116%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:03.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 279795 9.96 MB/sec execute 565 sec latency 38.209 ms 2024-04-27T01:18:03.766 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:03 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:03.766 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:03 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:03.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:03 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:03.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:03 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:04.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 280180 9.97 MB/sec execute 566 sec latency 44.697 ms 2024-04-27T01:18:04.848 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:04 smithi104 ceph-mon[63925]: pgmap v277: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.7 MiB/s wr, 498 op/s; 199/1650 objects degraded (12.061%) 2024-04-27T01:18:04.848 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:04 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:04.848 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:04 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:04.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:04 smithi135 ceph-mon[50158]: pgmap v277: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.7 MiB/s wr, 498 op/s; 199/1650 objects degraded (12.061%) 2024-04-27T01:18:04.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:04 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:04.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:04 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:05.690 INFO:teuthology.orchestra.run.smithi104.stdout: 5 280470 9.96 MB/sec execute 567 sec latency 44.024 ms 2024-04-27T01:18:06.339 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:18:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:18:06.160+0000 7f04d93437c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-27T01:18:06.647 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:06 smithi104 ceph-mon[63925]: pgmap v278: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.7 MiB/s wr, 462 op/s; 205/1632 objects degraded (12.561%) 2024-04-27T01:18:06.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 280747 9.96 MB/sec execute 568 sec latency 55.745 ms 2024-04-27T01:18:06.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:06 smithi135 ceph-mon[50158]: pgmap v278: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.7 MiB/s wr, 462 op/s; 205/1632 objects degraded (12.561%) 2024-04-27T01:18:07.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 280987 9.95 MB/sec execute 569 sec latency 49.873 ms 2024-04-27T01:18:07.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:07 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 205/1632 objects degraded (12.561%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:07.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:07 smithi104 ceph-mon[63925]: from='osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:18:07.717 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:07 smithi104 ceph-mon[63925]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:18:07.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:07 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 205/1632 objects degraded (12.561%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:07.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:07 smithi135 ceph-mon[50158]: from='osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:18:07.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:07 smithi135 ceph-mon[50158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:18:07.863 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:18:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:18:07.526+0000 7f04d93437c0 -1 osd.6 145 log_to_monitors true 2024-04-27T01:18:07.863 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:18:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:18:07.668+0000 7f04ca79b700 -1 osd.6 145 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:18:08.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281262 9.94 MB/sec execute 570 sec latency 105.914 ms 2024-04-27T01:18:08.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: pgmap v279: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 494 op/s; 208/1686 objects degraded (12.337%) 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: osdmap e148: 8 total, 7 up, 8 in 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:08.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:08.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:08.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: pgmap v279: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 344 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.2 MiB/s wr, 494 op/s; 208/1686 objects degraded (12.337%) 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: osdmap e148: 8 total, 7 up, 8 in 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:09.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:09.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:09.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:09.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:09.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:09.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281262 9.92 MB/sec execute 571 sec latency 1105.947 ms 2024-04-27T01:18:09.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: pgmap v281: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.4 MiB/s wr, 430 op/s; 207/1674 objects degraded (12.366%) 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: pgmap v282: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 4.8 MiB/s wr, 371 op/s; 207/1674 objects degraded (12.366%) 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858] boot 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:18:09.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:09 smithi104 ceph-mon[63925]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T01:18:10.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: pgmap v281: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.4 MiB/s wr, 430 op/s; 207/1674 objects degraded (12.366%) 2024-04-27T01:18:10.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: pgmap v282: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 4.8 MiB/s wr, 371 op/s; 207/1674 objects degraded (12.366%) 2024-04-27T01:18:10.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:10.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-27T01:18:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:18:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: osd.6 [v2:172.21.15.135:6816/2694381858,v1:172.21.15.135:6817/2694381858] boot 2024-04-27T01:18:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T01:18:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:18:10.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:09 smithi135 ceph-mon[50158]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T01:18:10.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281278 9.90 MB/sec execute 572 sec latency 2060.786 ms 2024-04-27T01:18:11.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:11 smithi104 ceph-mon[63925]: pgmap v285: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 208/1698 objects degraded (12.250%) 2024-04-27T01:18:11.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:11 smithi104 ceph-mon[63925]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T01:18:11.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281544 9.90 MB/sec execute 573 sec latency 78.842 ms 2024-04-27T01:18:12.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:11 smithi135 ceph-mon[50158]: pgmap v285: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 453 op/s; 208/1698 objects degraded (12.250%) 2024-04-27T01:18:12.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:11 smithi135 ceph-mon[50158]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T01:18:12.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:12 smithi104 ceph-mon[63925]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T01:18:12.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:12 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 208/1698 objects degraded (12.250%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:12.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:12 smithi104 ceph-mon[63925]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T01:18:12.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:12.685 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:12.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281656 9.88 MB/sec execute 574 sec latency 568.531 ms 2024-04-27T01:18:13.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:12 smithi135 ceph-mon[50158]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T01:18:13.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:12 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 208/1698 objects degraded (12.250%), 25 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:13.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:12 smithi135 ceph-mon[50158]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T01:18:13.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:13.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:13.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 281957 9.88 MB/sec execute 575 sec latency 702.319 ms 2024-04-27T01:18:13.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:13 smithi104 ceph-mon[63925]: pgmap v288: 97 pgs: 2 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 76 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.5 MiB/s wr, 413 op/s; 182/1590 objects degraded (11.447%); 704 KiB/s, 68 keys/s, 13 objects/s recovering 2024-04-27T01:18:13.736 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:13 smithi104 ceph-mon[63925]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T01:18:14.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:13 smithi135 ceph-mon[50158]: pgmap v288: 97 pgs: 2 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 76 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 3.5 MiB/s wr, 413 op/s; 182/1590 objects degraded (11.447%); 704 KiB/s, 68 keys/s, 13 objects/s recovering 2024-04-27T01:18:14.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:13 smithi135 ceph-mon[50158]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T01:18:14.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 282246 9.87 MB/sec execute 576 sec latency 902.187 ms 2024-04-27T01:18:15.254 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:15 smithi104 ceph-mon[63925]: pgmap v291: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.0 MiB/s wr, 277 op/s; 71/1509 objects degraded (4.705%); 1.5 MiB/s, 68 keys/s, 41 objects/s recovering 2024-04-27T01:18:15.254 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:15 smithi104 ceph-mon[63925]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T01:18:15.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:15 smithi135 ceph-mon[50158]: pgmap v291: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.0 MiB/s wr, 277 op/s; 71/1509 objects degraded (4.705%); 1.5 MiB/s, 68 keys/s, 41 objects/s recovering 2024-04-27T01:18:15.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:15 smithi135 ceph-mon[50158]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T01:18:15.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 282571 9.88 MB/sec execute 577 sec latency 51.943 ms 2024-04-27T01:18:16.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 282836 9.87 MB/sec execute 578 sec latency 94.764 ms 2024-04-27T01:18:17.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:17 smithi104 ceph-mon[63925]: pgmap v293: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.9 MiB/s wr, 198 op/s; 71/1512 objects degraded (4.696%); 738 KiB/s, 62 keys/s, 29 objects/s recovering 2024-04-27T01:18:17.309 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:17 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 71/1512 objects degraded (4.696%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:17 smithi135 ceph-mon[50158]: pgmap v293: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.9 MiB/s wr, 198 op/s; 71/1512 objects degraded (4.696%); 738 KiB/s, 62 keys/s, 29 objects/s recovering 2024-04-27T01:18:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:17 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 71/1512 objects degraded (4.696%), 3 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:17.687 INFO:teuthology.orchestra.run.smithi104.stdout: 5 283203 9.87 MB/sec execute 579 sec latency 34.137 ms 2024-04-27T01:18:18.340 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:18 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/1512 objects degraded (4.696%), 3 pgs degraded) 2024-04-27T01:18:18.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:18 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/1512 objects degraded (4.696%), 3 pgs degraded) 2024-04-27T01:18:18.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 283448 9.86 MB/sec execute 580 sec latency 55.730 ms 2024-04-27T01:18:19.372 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:19 smithi104 ceph-mon[63925]: pgmap v294: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.2 MiB/s wr, 407 op/s; 875 KiB/s, 30 objects/s recovering 2024-04-27T01:18:19.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:19 smithi135 ceph-mon[50158]: pgmap v294: 97 pgs: 97 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.2 MiB/s wr, 407 op/s; 875 KiB/s, 30 objects/s recovering 2024-04-27T01:18:19.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 283780 9.87 MB/sec execute 581 sec latency 36.920 ms 2024-04-27T01:18:20.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 284035 9.87 MB/sec execute 582 sec latency 54.201 ms 2024-04-27T01:18:21.450 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:21 smithi104 ceph-mon[63925]: pgmap v295: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.1 MiB/s wr, 347 op/s; 667 KiB/s, 23 objects/s recovering 2024-04-27T01:18:21.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:21 smithi135 ceph-mon[50158]: pgmap v295: 97 pgs: 97 active+clean; 332 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.1 MiB/s wr, 347 op/s; 667 KiB/s, 23 objects/s recovering 2024-04-27T01:18:21.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 284357 9.87 MB/sec execute 583 sec latency 69.008 ms 2024-04-27T01:18:22.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 284628 9.86 MB/sec execute 584 sec latency 67.659 ms 2024-04-27T01:18:23.240 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:23 smithi104 ceph-mon[63925]: pgmap v296: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 393 op/s; 486 KiB/s, 13 objects/s recovering 2024-04-27T01:18:23.240 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:23 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:23.315 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:18:23.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:23 smithi135 ceph-mon[50158]: pgmap v296: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 393 op/s; 486 KiB/s, 13 objects/s recovering 2024-04-27T01:18:23.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:23 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:23.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 284819 9.85 MB/sec execute 585 sec latency 65.372 ms 2024-04-27T01:18:24.168 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (9m) 2m ago 18m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (5m) 2m ago 18m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (5m) 21s ago 17m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 2m ago 17m - - 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (12m) 2m ago 12m 41.2M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (12m) 21s ago 12m 17.7M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (7m) 21s ago 17m 452M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (6m) 2m ago 19m 51.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (6m) 21s ago 17m 45.8M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (11m) 2m ago 11m 274M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (11m) 21s ago 11m 85.8M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (8m) 2m ago 17m 20.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:18:24.169 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 21s ago 17m 17.7M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (4m) 2m ago 16m 336M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (3m) 2m ago 16m 296M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (3m) 2m ago 15m 230M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (2m) 2m ago 15m 15.2M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (98s) 21s ago 14m 292M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (62s) 21s ago 14m 250M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (26s) 21s ago 14m 11.7M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (13m) 21s ago 13m 635M 2704M 17.2.0 e1d6a67b021e 776cbc1f18ec 2024-04-27T01:18:24.170 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (7m) 2m ago 17m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:18:24.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:24 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:24.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:24 smithi104 ceph-mon[63925]: Upgrade: osd.7 is safe to restart 2024-04-27T01:18:24.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:24 smithi104 ceph-mon[63925]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:24.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:24 smithi104 ceph-mon[63925]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:24.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:24 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-27T01:18:24.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:24 smithi135 ceph-mon[50158]: Upgrade: osd.7 is safe to restart 2024-04-27T01:18:24.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:24 smithi135 ceph-mon[50158]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:24.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:24 smithi135 ceph-mon[50158]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:24.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 285057 9.84 MB/sec execute 586 sec latency 92.854 ms 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 7 2024-04-27T01:18:24.748 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 11 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:18:24.749 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:18:25.208 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:18:25.208 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr" 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "13/23 daemons upgraded", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading osd daemons", 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:18:25.209 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:18:25.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: pgmap v297: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 411 op/s; 212 KiB/s, 7 objects/s recovering 2024-04-27T01:18:25.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:25.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/592893434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:25.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:25.282 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:18:25.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:25 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:25.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: pgmap v297: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 411 op/s; 212 KiB/s, 7 objects/s recovering 2024-04-27T01:18:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/592893434' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:18:25.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:25 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:25.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 285390 9.84 MB/sec execute 587 sec latency 85.217 ms 2024-04-27T01:18:25.891 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:18:25.892 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:18:25.892 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:18:26.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:26 smithi104 ceph-mon[63925]: Upgrade: Updating osd.7 2024-04-27T01:18:26.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:26 smithi104 ceph-mon[63925]: Deploying daemon osd.7 on smithi135 2024-04-27T01:18:26.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:26 smithi104 ceph-mon[63925]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:26.318 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:26 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1512663970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:18:26.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:26 smithi135 ceph-mon[50158]: Upgrade: Updating osd.7 2024-04-27T01:18:26.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:26 smithi135 ceph-mon[50158]: Deploying daemon osd.7 on smithi135 2024-04-27T01:18:26.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:26 smithi135 ceph-mon[50158]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:26.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:26 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1512663970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:18:26.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 285693 9.83 MB/sec execute 588 sec latency 56.013 ms 2024-04-27T01:18:27.343 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:27 smithi104 ceph-mon[63925]: pgmap v298: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 353 op/s; 179 KiB/s, 5 objects/s recovering 2024-04-27T01:18:27.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:27 smithi135 ceph-mon[50158]: pgmap v298: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.7 MiB/s wr, 353 op/s; 179 KiB/s, 5 objects/s recovering 2024-04-27T01:18:27.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 286035 9.83 MB/sec execute 589 sec latency 54.022 ms 2024-04-27T01:18:28.651 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:28.651 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:28.651 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:28.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 286368 9.83 MB/sec execute 590 sec latency 48.186 ms 2024-04-27T01:18:28.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:28.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:28.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:29.599 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:29 smithi135 systemd[1]: Stopping Ceph osd.7 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:18:29.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 286764 9.83 MB/sec execute 591 sec latency 44.266 ms 2024-04-27T01:18:29.862 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:29 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[37359]: 2024-04-27T01:18:29.610+0000 7fa42fe7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:18:29.862 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:29 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[37359]: 2024-04-27T01:18:29.610+0000 7fa42fe7c700 -1 osd.7 155 *** Got signal Terminated *** 2024-04-27T01:18:29.862 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:29 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[37359]: 2024-04-27T01:18:29.610+0000 7fa42fe7c700 -1 osd.7 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:18:30.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:29 smithi104 ceph-mon[63925]: pgmap v299: 97 pgs: 97 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.6 MiB/s wr, 485 op/s; 177 KiB/s, 5 objects/s recovering 2024-04-27T01:18:30.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:29 smithi104 ceph-mon[63925]: osd.7 marked itself down and dead 2024-04-27T01:18:30.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:29 smithi135 ceph-mon[50158]: pgmap v299: 97 pgs: 97 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.6 MiB/s wr, 485 op/s; 177 KiB/s, 5 objects/s recovering 2024-04-27T01:18:30.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:29 smithi135 ceph-mon[50158]: osd.7 marked itself down and dead 2024-04-27T01:18:30.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 286834 9.82 MB/sec execute 592 sec latency 823.843 ms 2024-04-27T01:18:30.854 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:30 smithi135 podman[61391]: 2024-04-27 01:18:30.598792159 +0000 UTC m=+1.139371650 container died 776cbc1f18eca52ac950a8e0994fc35e19e05789ebbf4568256b7f1ff34d27b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git) 2024-04-27T01:18:31.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:30 smithi135 ceph-mon[50158]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:18:31.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:30 smithi135 ceph-mon[50158]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-27T01:18:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:30 smithi135 ceph-mon[50158]: osdmap e156: 8 total, 7 up, 8 in 2024-04-27T01:18:31.113 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:30 smithi135 podman[61391]: 2024-04-27 01:18:30.957705233 +0000 UTC m=+1.498284716 container cleanup 776cbc1f18eca52ac950a8e0994fc35e19e05789ebbf4568256b7f1ff34d27b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , vcs-type=git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, architecture=x86_64, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container) 2024-04-27T01:18:31.113 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:30 smithi135 bash[61391]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7 2024-04-27T01:18:31.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:30 smithi104 ceph-mon[63925]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-27T01:18:31.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:30 smithi104 ceph-mon[63925]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-27T01:18:31.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:30 smithi104 ceph-mon[63925]: osdmap e156: 8 total, 7 up, 8 in 2024-04-27T01:18:31.612 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:31 smithi135 podman[61409]: 2024-04-27 01:18:31.28422738 +0000 UTC m=+0.685647991 container remove 776cbc1f18eca52ac950a8e0994fc35e19e05789ebbf4568256b7f1ff34d27b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, version=8, distribution-scope=public, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8) 2024-04-27T01:18:31.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 287064 9.81 MB/sec execute 593 sec latency 1056.323 ms 2024-04-27T01:18:31.916 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:31 smithi135 podman[61496]: 2024-04-27 01:18:31.563131631 +0000 UTC m=+0.020419891 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:18:31.916 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:31 smithi135 podman[61496]: 2024-04-27 01:18:31.860524222 +0000 UTC m=+0.317812511 container create da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:18:31.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:31 smithi104 ceph-mon[63925]: pgmap v301: 97 pgs: 12 stale+active+clean, 85 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 415 op/s 2024-04-27T01:18:31.924 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:31 smithi104 ceph-mon[63925]: osdmap e157: 8 total, 7 up, 8 in 2024-04-27T01:18:32.183 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:31 smithi135 ceph-mon[50158]: pgmap v301: 97 pgs: 12 stale+active+clean, 85 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 415 op/s 2024-04-27T01:18:32.183 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:31 smithi135 ceph-mon[50158]: osdmap e157: 8 total, 7 up, 8 in 2024-04-27T01:18:32.184 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:32 smithi135 podman[61496]: 2024-04-27 01:18:32.105325755 +0000 UTC m=+0.562613999 container init da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-27T01:18:32.184 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:32 smithi135 podman[61496]: 2024-04-27 01:18:32.1097305 +0000 UTC m=+0.567018749 container start da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-27T01:18:32.184 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:32 smithi135 podman[61496]: 2024-04-27 01:18:32.196927843 +0000 UTC m=+0.654216097 container attach da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:18:32.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 287788 9.84 MB/sec execute 594 sec latency 52.866 ms 2024-04-27T01:18:32.851 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:32 smithi135 podman[61496]: 2024-04-27 01:18:32.469549731 +0000 UTC m=+0.926837976 container died da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-27T01:18:32.933 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:32 smithi104 ceph-mon[63925]: pgmap v303: 97 pgs: 9 active+undersized+degraded, 3 active+undersized, 7 peering, 6 stale+active+clean, 72 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 419 op/s; 60/1641 objects degraded (3.656%) 2024-04-27T01:18:33.101 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:32 smithi135 ceph-mon[50158]: pgmap v303: 97 pgs: 9 active+undersized+degraded, 3 active+undersized, 7 peering, 6 stale+active+clean, 72 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 419 op/s; 60/1641 objects degraded (3.656%) 2024-04-27T01:18:33.101 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61521]: 2024-04-27 01:18:33.101830926 +0000 UTC m=+0.626568710 container remove da4404d0face3e8c2087ae0cfffecb096d57801607e09ba3523994d131371a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T01:18:33.362 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-27T01:18:33.362 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 systemd[1]: Stopped Ceph osd.7 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:18:33.362 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 systemd[1]: ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.7.service: Consumed 7min 31.306s CPU time. 2024-04-27T01:18:33.631 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 systemd[1]: Starting Ceph osd.7 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:18:33.632 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61616]: 2024-04-27 01:18:33.526121683 +0000 UTC m=+0.022859774 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:18:33.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 288175 9.84 MB/sec execute 595 sec latency 51.406 ms 2024-04-27T01:18:33.922 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61616]: 2024-04-27 01:18:33.644999506 +0000 UTC m=+0.141737580 container create 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-27T01:18:33.922 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61616]: 2024-04-27 01:18:33.881253469 +0000 UTC m=+0.377991543 container init 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb) 2024-04-27T01:18:33.922 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61616]: 2024-04-27 01:18:33.885803027 +0000 UTC m=+0.382541106 container start 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-27T01:18:33.951 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:33 smithi104 ceph-mon[63925]: Health check failed: Degraded data redundancy: 60/1641 objects degraded (3.656%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:34.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:33 smithi135 ceph-mon[50158]: Health check failed: Degraded data redundancy: 60/1641 objects degraded (3.656%), 9 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:34.362 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:33 smithi135 podman[61616]: 2024-04-27 01:18:33.989850569 +0000 UTC m=+0.486588661 container attach 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 bash[61616]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-27T01:18:34.667 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-27T01:18:34.668 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-27T01:18:34.668 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate[61631]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-27T01:18:34.668 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:34 smithi135 podman[61616]: 2024-04-27 01:18:34.462937234 +0000 UTC m=+0.959675325 container died 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-27T01:18:34.688 INFO:teuthology.orchestra.run.smithi104.stdout: 5 288448 9.83 MB/sec execute 596 sec latency 64.243 ms 2024-04-27T01:18:34.972 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:34 smithi104 ceph-mon[63925]: pgmap v304: 97 pgs: 27 active+undersized+degraded, 8 active+undersized, 7 peering, 55 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.3 MiB/s wr, 550 op/s; 166/1680 objects degraded (9.881%) 2024-04-27T01:18:35.337 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:34 smithi135 ceph-mon[50158]: pgmap v304: 97 pgs: 27 active+undersized+degraded, 8 active+undersized, 7 peering, 55 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.3 MiB/s wr, 550 op/s; 166/1680 objects degraded (9.881%) 2024-04-27T01:18:35.337 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:35 smithi135 podman[61758]: 2024-04-27 01:18:35.119980746 +0000 UTC m=+0.651267143 container remove 36b014a160be5789a586f333a0e57d8b2ddd8787a1b1f63efac6b96f7bf11a9e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:18:35.612 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:35 smithi135 podman[61814]: 2024-04-27 01:18:35.253727192 +0000 UTC m=+0.021530583 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:18:35.689 INFO:teuthology.orchestra.run.smithi104.stdout: 5 288791 9.83 MB/sec execute 597 sec latency 68.007 ms 2024-04-27T01:18:36.140 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:35 smithi135 podman[61814]: 2024-04-27 01:18:35.888153713 +0000 UTC m=+0.655957085 container create 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T01:18:36.140 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 podman[61814]: 2024-04-27 01:18:36.124127781 +0000 UTC m=+0.891931154 container init 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:18:36.140 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 podman[61814]: 2024-04-27 01:18:36.128639427 +0000 UTC m=+0.896442807 container start 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:18:36.140 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 ceph-osd[61866]: -- 172.21.15.135:0/4089309864 <== mon.1 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f3b8f1680 con 0x564f3b8bac00 2024-04-27T01:18:36.612 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 bash[61814]: 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c 2024-04-27T01:18:36.612 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 systemd[1]: Started Ceph osd.7 for 38f5b578-0431-11ef-bc93-c7b262605968. 2024-04-27T01:18:36.689 INFO:teuthology.orchestra.run.smithi104.stdout: 5 289074 9.82 MB/sec execute 598 sec latency 88.701 ms 2024-04-27T01:18:37.112 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:36 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:18:36.671+0000 7f179b5bf7c0 -1 Falling back to public interface 2024-04-27T01:18:37.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:37 smithi104 ceph-mon[63925]: pgmap v305: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 368 op/s; 255/1683 objects degraded (15.152%) 2024-04-27T01:18:37.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:37.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:37.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:37.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:37 smithi135 ceph-mon[50158]: pgmap v305: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s wr, 368 op/s; 255/1683 objects degraded (15.152%) 2024-04-27T01:18:37.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:37.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:37.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:37.689 INFO:teuthology.orchestra.run.smithi104.stdout: 5 289406 9.82 MB/sec execute 599 sec latency 41.554 ms 2024-04-27T01:18:38.575 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:38 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 255/1650 objects degraded (15.455%), 34 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:38.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:38 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 255/1650 objects degraded (15.455%), 34 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:38.689 INFO:teuthology.orchestra.run.smithi104.stdout: 5 cleanup 600 sec 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: 0 cleanup 600 sec 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Operation Count AvgLat MaxLat 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: ---------------------------------------- 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: NTCreateX 189423 4.520 2718.510 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Close 139060 0.132 17.491 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Rename 8023 9.597 1048.759 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Unlink 38329 7.115 2710.683 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Qpathinfo 171716 1.134 106.684 2024-04-27T01:18:38.921 INFO:teuthology.orchestra.run.smithi104.stdout: Qfileinfo 29895 0.007 1.880 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: Qfsinfo 31488 0.611 14.723 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: Sfileinfo 15462 13.301 2714.920 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: Find 66342 0.933 17.082 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: WriteX 93524 13.765 2719.444 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: ReadX 296596 0.010 7.831 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: LockX 614 0.393 4.310 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: UnlockX 614 0.340 3.749 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: Flush 13286 0.002 0.069 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:18:38.922 INFO:teuthology.orchestra.run.smithi104.stdout:Throughput 9.81907 MB/sec 5 clients 5 procs max_latency=2719.450 ms 2024-04-27T01:18:38.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-04-27T01:18:38.974 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-04-27T01:18:39.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-04-27T01:18:39.114 INFO:teuthology.orchestra.run.smithi104.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-27T01:18:39.115 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:18:39.115 INFO:teuthology.orchestra.run.smithi104.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-04-27T01:18:39.115 INFO:teuthology.orchestra.run.smithi104.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-27T01:18:39.216 INFO:teuthology.orchestra.run.smithi104.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-27T01:18:39.216 INFO:teuthology.orchestra.run.smithi104.stdout:releasing clients 2024-04-27T01:18:39.346 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:39 smithi104 ceph-mon[63925]: pgmap v306: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 604 op/s; 255/1650 objects degraded (15.455%) 2024-04-27T01:18:39.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:39 smithi135 ceph-mon[50158]: pgmap v306: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.6 MiB/s wr, 604 op/s; 255/1650 objects degraded (15.455%) 2024-04-27T01:18:41.216 INFO:teuthology.orchestra.run.smithi104.stdout: 5 366 21.85 MB/sec execute 1 sec latency 42.140 ms 2024-04-27T01:18:41.451 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:41 smithi104 ceph-mon[63925]: pgmap v307: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 330 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 508 op/s; 253/1617 objects degraded (15.646%) 2024-04-27T01:18:41.451 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:41 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:41.451 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:41 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:41.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:41 smithi135 ceph-mon[50158]: pgmap v307: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 330 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 508 op/s; 253/1617 objects degraded (15.646%) 2024-04-27T01:18:41.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:41 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:41.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:41 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:42.216 INFO:teuthology.orchestra.run.smithi104.stdout: 5 475 19.77 MB/sec execute 2 sec latency 83.033 ms 2024-04-27T01:18:42.900 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:42.900 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:42.900 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:42.900 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:43.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:43.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:43.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:43.059 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:43.216 INFO:teuthology.orchestra.run.smithi104.stdout: 5 723 20.93 MB/sec execute 3 sec latency 34.829 ms 2024-04-27T01:18:43.931 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:43 smithi135 ceph-mon[50158]: pgmap v308: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 8.0 MiB/s wr, 500 op/s; 250/1548 objects degraded (16.150%) 2024-04-27T01:18:44.074 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:43 smithi104 ceph-mon[63925]: pgmap v308: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 8.0 MiB/s wr, 500 op/s; 250/1548 objects degraded (16.150%) 2024-04-27T01:18:44.216 INFO:teuthology.orchestra.run.smithi104.stdout: 5 1122 17.81 MB/sec execute 4 sec latency 29.007 ms 2024-04-27T01:18:45.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:44 smithi104 ceph-mon[63925]: pgmap v309: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 11 MiB/s wr, 530 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:45.088 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:44 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 230/1410 objects degraded (16.312%), 34 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:45.216 INFO:teuthology.orchestra.run.smithi104.stdout: 5 cleanup 5 sec 2024-04-27T01:18:45.246 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:44 smithi135 ceph-mon[50158]: pgmap v309: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 11 MiB/s wr, 530 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:45.246 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:44 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 230/1410 objects degraded (16.312%), 34 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:45.360 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c91900000027' failed - Device or resource busy 2024-04-27T01:18:45.360 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c91d00000043' failed - Device or resource busy 2024-04-27T01:18:45.361 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c90e00000042' failed - Device or resource busy 2024-04-27T01:18:45.362 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000ca050000003c' failed - Device or resource busy 2024-04-27T01:18:45.362 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000ca0200000040' failed - Device or resource busy 2024-04-27T01:18:45.362 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000ca080000002d' failed - Device or resource busy 2024-04-27T01:18:45.363 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000c91b00000026' failed - Device or resource busy 2024-04-27T01:18:45.363 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000c9160000001d' failed - Device or resource busy 2024-04-27T01:18:45.363 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000ca0600000030' failed - Device or resource busy 2024-04-27T01:18:45.364 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000c92100000047' failed - Device or resource busy 2024-04-27T01:18:45.369 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client2/~dmtmp/PARADOX/.nfs000001000000c91100000044' failed - Device or resource busy 2024-04-27T01:18:45.370 INFO:teuthology.orchestra.run.smithi104.stdout:[1534] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c9140000001e' failed - Device or resource busy 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: 0 cleanup 5 sec 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: Operation Count AvgLat MaxLat 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: ---------------------------------------- 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: NTCreateX 1037 6.666 83.029 2024-04-27T01:18:45.426 INFO:teuthology.orchestra.run.smithi104.stdout: Close 877 6.701 75.885 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Rename 50 10.246 33.011 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Unlink 103 3.756 20.539 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Qpathinfo 663 1.507 19.769 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Qfileinfo 384 17.032 68.556 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Qfsinfo 165 0.464 3.153 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Sfileinfo 170 11.431 24.840 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Find 281 0.758 2.926 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: WriteX 1697 0.022 0.106 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: ReadX 1229 0.016 0.480 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: Flush 127 12.289 23.197 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:18:45.427 INFO:teuthology.orchestra.run.smithi104.stdout:Throughput 17.8075 MB/sec 5 clients 5 procs max_latency=83.033 ms 2024-04-27T01:18:45.428 DEBUG:teuthology.parallel:result is None 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: pgmap v310: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.9 MiB/s wr, 448 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: pgmap v311: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 516 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all osd 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-27T01:18:47.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: Upgrade: Setting require_osd_release to 19 squid 2024-04-27T01:18:47.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: pgmap v310: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.9 MiB/s wr, 448 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: pgmap v311: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 516 op/s; 230/1410 objects degraded (16.312%) 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all osd 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-27T01:18:47.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: Upgrade: Setting require_osd_release to 19 squid 2024-04-27T01:18:47.165 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-27T01:18:47.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:47 smithi104 ceph-mon[63925]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-27T01:18:47.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-27T01:18:47.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:47 smithi104 ceph-mon[63925]: osdmap e158: 8 total, 7 up, 8 in 2024-04-27T01:18:47.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi104.wfnrbv"]}]: dispatch 2024-04-27T01:18:47.793 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:47 smithi104 ceph-mon[63925]: Upgrade: It appears safe to stop mds.foofs.smithi104.wfnrbv 2024-04-27T01:18:47.862 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:18:47.514+0000 7f179b5bf7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-27T01:18:47.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-mon[50158]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-27T01:18:47.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-27T01:18:47.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-mon[50158]: osdmap e158: 8 total, 7 up, 8 in 2024-04-27T01:18:47.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi104.wfnrbv"]}]: dispatch 2024-04-27T01:18:47.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:47 smithi135 ceph-mon[50158]: Upgrade: It appears safe to stop mds.foofs.smithi104.wfnrbv 2024-04-27T01:18:48.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:48 smithi104 ceph-mon[63925]: pgmap v313: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 17 MiB/s wr, 550 op/s; 225/1455 objects degraded (15.464%) 2024-04-27T01:18:49.020 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:48 smithi135 ceph-mon[50158]: pgmap v313: 97 pgs: 34 active+undersized+degraded, 8 active+undersized, 55 active+clean; 326 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 17 MiB/s wr, 550 op/s; 225/1455 objects degraded (15.464%) 2024-04-27T01:18:49.362 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:18:49.033+0000 7f179b5bf7c0 -1 osd.7 155 log_to_monitors true 2024-04-27T01:18:49.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:49 smithi104 ceph-mon[63925]: from='osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:18:49.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:49 smithi104 ceph-mon[63925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:18:49.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:49.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:18:49.906 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-mon[50158]: from='osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:18:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-mon[50158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:18:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi104.wfnrbv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:18:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:50.612 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:18:50.160+0000 7f1788a0f700 -1 osd.7 155 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:18:50.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: Upgrade: Updating mds.foofs.smithi104.wfnrbv 2024-04-27T01:18:50.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: Deploying daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:18:50.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: pgmap v314: 97 pgs: 33 active+undersized+degraded, 9 active+undersized, 55 active+clean; 304 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 482 op/s; 208/1287 objects degraded (16.162%) 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: osdmap e159: 8 total, 7 up, 8 in 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: from='osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 208/1287 objects degraded (16.162%), 33 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:50.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:50 smithi104 ceph-mon[63925]: from='osd.7 ' entity='osd.7' 2024-04-27T01:18:51.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: Upgrade: Updating mds.foofs.smithi104.wfnrbv 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: Deploying daemon mds.foofs.smithi104.wfnrbv on smithi104 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: pgmap v314: 97 pgs: 33 active+undersized+degraded, 9 active+undersized, 55 active+clean; 304 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 482 op/s; 208/1287 objects degraded (16.162%) 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: osdmap e159: 8 total, 7 up, 8 in 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: from='osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 208/1287 objects degraded (16.162%), 33 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:51.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:50 smithi135 ceph-mon[50158]: from='osd.7 ' entity='osd.7' 2024-04-27T01:18:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:18:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292] boot 2024-04-27T01:18:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: osdmap e160: 8 total, 8 up, 8 in 2024-04-27T01:18:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:18:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: osdmap e161: 8 total, 8 up, 8 in 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Standby daemon mds.foofs.smithi135.brxmmu assigned to filesystem foofs as rank 0 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T01:18:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:51 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:replay} 2024-04-27T01:18:52.163 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:18:51.778+0000 7f1c192f7700 -1 bdev(0x55fb8410c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 9 2024-04-27T01:18:52.163 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:18:51.780+0000 7f1c152ef700 -1 bdev(0x55fb8410c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 8 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[63921]: 2024-04-27T01:18:51.679+0000 7f8042171700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: osd.7 [v2:172.21.15.135:6824/67407292,v1:172.21.15.135:6825/67407292] boot 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: osdmap e160: 8 total, 8 up, 8 in 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:18:52.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: osdmap e161: 8 total, 8 up, 8 in 2024-04-27T01:18:52.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Standby daemon mds.foofs.smithi135.brxmmu assigned to filesystem foofs as rank 0 2024-04-27T01:18:52.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T01:18:52.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:18:52.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T01:18:52.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:51 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:replay} 2024-04-27T01:18:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:52 smithi135 ceph-mon[50158]: pgmap v318: 97 pgs: 27 active+undersized+degraded, 15 active+undersized, 55 active+clean; 273 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 307 op/s; 133/990 objects degraded (13.434%) 2024-04-27T01:18:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:52 smithi135 ceph-mon[50158]: osdmap e162: 8 total, 8 up, 8 in 2024-04-27T01:18:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:52 smithi104 ceph-mon[63925]: pgmap v318: 97 pgs: 27 active+undersized+degraded, 15 active+undersized, 55 active+clean; 273 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.1 MiB/s wr, 307 op/s; 133/990 objects degraded (13.434%) 2024-04-27T01:18:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:52 smithi104 ceph-mon[63925]: osdmap e162: 8 total, 8 up, 8 in 2024-04-27T01:18:54.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:53 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:18:54.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:53 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:18:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: pgmap v320: 97 pgs: 7 remapped+peering, 1 activating, 1 active+recovering+degraded, 1 active+undersized+remapped, 9 active+undersized+degraded, 9 active+undersized, 69 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 58 KiB/s wr, 78 op/s; 38/453 objects degraded (8.389%); 95 KiB/s, 10 objects/s recovering 2024-04-27T01:18:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: reconnect by client.24337 172.21.15.135:0/3425316284 after 0.000999978 2024-04-27T01:18:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: reconnect by client.14552 172.21.15.104:0/3837531920 after 0.000999978 2024-04-27T01:18:55.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: osdmap e163: 8 total, 8 up, 8 in 2024-04-27T01:18:55.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:reconnect 2024-04-27T01:18:55.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:54 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:reconnect} 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: pgmap v320: 97 pgs: 7 remapped+peering, 1 activating, 1 active+recovering+degraded, 1 active+undersized+remapped, 9 active+undersized+degraded, 9 active+undersized, 69 active+clean; 224 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 58 KiB/s wr, 78 op/s; 38/453 objects degraded (8.389%); 95 KiB/s, 10 objects/s recovering 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: reconnect by client.24337 172.21.15.135:0/3425316284 after 0.000999978 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: reconnect by client.14552 172.21.15.104:0/3837531920 after 0.000999978 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: osdmap e163: 8 total, 8 up, 8 in 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:reconnect 2024-04-27T01:18:55.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:54 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:reconnect} 2024-04-27T01:18:56.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:55 smithi135 ceph-mon[50158]: osdmap e164: 8 total, 8 up, 8 in 2024-04-27T01:18:56.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:55 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:rejoin 2024-04-27T01:18:56.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:55 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:rejoin} 2024-04-27T01:18:56.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:55 smithi135 ceph-mon[50158]: daemon mds.foofs.smithi135.brxmmu is now active in filesystem foofs as rank 0 2024-04-27T01:18:56.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:55 smithi104 ceph-mon[63925]: osdmap e164: 8 total, 8 up, 8 in 2024-04-27T01:18:56.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:55 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:rejoin 2024-04-27T01:18:56.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:55 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi135.brxmmu=up:rejoin} 2024-04-27T01:18:56.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:55 smithi104 ceph-mon[63925]: daemon mds.foofs.smithi135.brxmmu is now active in filesystem foofs as rank 0 2024-04-27T01:18:56.248 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (9m) 2m ago 18m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (5m) 2m ago 18m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (5m) 16s ago 18m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 2m ago 18m - - 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (12m) 2m ago 12m 41.2M - 17.2.0 e1d6a67b021e 0f841c4e55c6 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (12m) 16s ago 12m 17.9M - 17.2.0 e1d6a67b021e 3f2c89759059 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (7m) 16s ago 18m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (7m) 2m ago 20m 51.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (6m) 16s ago 17m 47.3M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:18:56.913 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (12m) 2m ago 12m 274M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (12m) 16s ago 12m 85.9M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (9m) 2m ago 18m 20.5M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (8m) 16s ago 17m 17.5M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (5m) 2m ago 17m 336M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (4m) 2m ago 16m 296M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (3m) 2m ago 16m 230M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (2m) 2m ago 15m 15.2M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (2m) 16s ago 15m 332M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (95s) 16s ago 15m 296M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (59s) 16s ago 14m 290M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (21s) 16s ago 14m 13.5M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:18:56.914 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (7m) 2m ago 17m 68.4M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:18:57.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:56 smithi135 ceph-mon[50158]: pgmap v323: 97 pgs: 2 peering, 5 remapped+peering, 1 activating, 1 active+recovering+degraded, 1 active+undersized+degraded, 2 active+undersized, 85 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 58 KiB/s wr, 58 op/s; 6/426 objects degraded (1.408%); 94 KiB/s, 64 objects/s recovering 2024-04-27T01:18:57.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:56 smithi135 ceph-mon[50158]: Health check update: Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:57.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:56 smithi135 ceph-mon[50158]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T01:18:57.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:56 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:active 2024-04-27T01:18:57.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:56 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi135.brxmmu=up:active} 2024-04-27T01:18:57.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:56 smithi104 ceph-mon[63925]: pgmap v323: 97 pgs: 2 peering, 5 remapped+peering, 1 activating, 1 active+recovering+degraded, 1 active+undersized+degraded, 2 active+undersized, 85 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 58 KiB/s wr, 58 op/s; 6/426 objects degraded (1.408%); 94 KiB/s, 64 objects/s recovering 2024-04-27T01:18:57.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:56 smithi104 ceph-mon[63925]: Health check update: Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded (PG_DEGRADED) 2024-04-27T01:18:57.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:56 smithi104 ceph-mon[63925]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T01:18:57.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:56 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.135:6832/2651278440,v1:172.21.15.135:6833/2651278440] up:active 2024-04-27T01:18:57.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:56 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi135.brxmmu=up:active} 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:57.315 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 12 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:18:57.316 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:18:57.656 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:18:57.656 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "osd" 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "14/23 daemons upgraded", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading mds daemons", 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:18:57.657 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:18:57.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:57.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:57.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:57.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2817086624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:57.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:57.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:58.063 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s); insufficient standby MDS daemons available; Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded 2024-04-27T01:18:58.063 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout: have 0; want 1 more 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.4 is active+recovering+degraded, acting [1,0,7] 2024-04-27T01:18:58.064 INFO:teuthology.orchestra.run.smithi104.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-27T01:18:58.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:58.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:58.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:58.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2817086624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:18:58.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:18:58.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:58.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:58 smithi104 ceph-mon[63925]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:58.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:58 smithi104 ceph-mon[63925]: pgmap v324: 97 pgs: 2 peering, 95 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 39 KiB/s wr, 45 op/s; 2.0 MiB/s, 71 objects/s recovering 2024-04-27T01:18:58.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:58 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/697425003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:18:58.850 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:58 smithi104 ceph-mon[63925]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded) 2024-04-27T01:18:59.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:58 smithi135 ceph-mon[50158]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:18:59.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:58 smithi135 ceph-mon[50158]: pgmap v324: 97 pgs: 2 peering, 95 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 33 MiB/s rd, 39 KiB/s wr, 45 op/s; 2.0 MiB/s, 71 objects/s recovering 2024-04-27T01:18:59.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:58 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/697425003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:18:59.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:58 smithi135 ceph-mon[50158]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/426 objects degraded (1.408%), 2 pgs degraded) 2024-04-27T01:18:59.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:59 smithi104 ceph-mon[63925]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T01:18:59.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:59 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:boot 2024-04-27T01:18:59.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:59 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi135.brxmmu=up:active} 1 up:standby 2024-04-27T01:18:59.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:18:59.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:18:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:59 smithi135 ceph-mon[50158]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T01:19:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:59 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:boot 2024-04-27T01:19:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:59 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi135.brxmmu=up:active} 1 up:standby 2024-04-27T01:19:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:19:00.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:18:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:00.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:00 smithi104 ceph-mon[63925]: pgmap v325: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 291 B/s wr, 12 op/s; 1.7 MiB/s, 61 objects/s recovering 2024-04-27T01:19:00.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:00 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:00.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:00 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:01.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:00 smithi135 ceph-mon[50158]: pgmap v325: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 291 B/s wr, 12 op/s; 1.7 MiB/s, 61 objects/s recovering 2024-04-27T01:19:01.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:00 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:01.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:00 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:03.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:02 smithi135 ceph-mon[50158]: pgmap v326: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 255 B/s wr, 10 op/s; 1.5 MiB/s, 48 objects/s recovering 2024-04-27T01:19:03.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:02 smithi104 ceph-mon[63925]: pgmap v326: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 255 B/s wr, 10 op/s; 1.5 MiB/s, 48 objects/s recovering 2024-04-27T01:19:05.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:04 smithi135 ceph-mon[50158]: pgmap v327: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 7.2 KiB/s wr, 11 op/s; 1.2 MiB/s, 39 objects/s recovering 2024-04-27T01:19:05.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:04 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:05.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:04 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:05.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:04 smithi104 ceph-mon[63925]: pgmap v327: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 7.2 KiB/s wr, 11 op/s; 1.2 MiB/s, 39 objects/s recovering 2024-04-27T01:19:05.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:04 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:05.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:04 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:06 smithi135 ceph-mon[50158]: pgmap v328: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 6.6 KiB/s wr, 10 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-04-27T01:19:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:07.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:06 smithi104 ceph-mon[63925]: pgmap v328: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 6.6 KiB/s wr, 10 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-04-27T01:19:07.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:07.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:09.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:08 smithi104 ceph-mon[63925]: pgmap v329: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 MiB/s rd, 7.2 KiB/s wr, 8 op/s; 1014 KiB/s, 14 objects/s recovering 2024-04-27T01:19:09.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:09.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:08 smithi135 ceph-mon[50158]: pgmap v329: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.7 MiB/s rd, 7.2 KiB/s wr, 8 op/s; 1014 KiB/s, 14 objects/s recovering 2024-04-27T01:19:09.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: pgmap v330: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:11.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.164 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:10 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi135.brxmmu"]}]: dispatch 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: pgmap v330: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.217 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:11.218 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:10 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi135.brxmmu"]}]: dispatch 2024-04-27T01:19:12.028 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:11 smithi135 ceph-mon[50158]: Detected new or changed devices on smithi104 2024-04-27T01:19:12.029 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:11 smithi135 ceph-mon[50158]: pgmap v331: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 8.1 KiB/s wr, 4 op/s 2024-04-27T01:19:12.029 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:11 smithi135 ceph-mon[50158]: pgmap v332: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 10 KiB/s wr, 4 op/s 2024-04-27T01:19:12.029 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:11 smithi135 ceph-mon[50158]: Upgrade: It appears safe to stop mds.foofs.smithi135.brxmmu 2024-04-27T01:19:12.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:11 smithi104 ceph-mon[63925]: Detected new or changed devices on smithi104 2024-04-27T01:19:12.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:11 smithi104 ceph-mon[63925]: pgmap v331: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 8.1 KiB/s wr, 4 op/s 2024-04-27T01:19:12.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:11 smithi104 ceph-mon[63925]: pgmap v332: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 10 KiB/s wr, 4 op/s 2024-04-27T01:19:12.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:11 smithi104 ceph-mon[63925]: Upgrade: It appears safe to stop mds.foofs.smithi135.brxmmu 2024-04-27T01:19:13.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: Upgrade: Updating mds.foofs.smithi135.brxmmu 2024-04-27T01:19:13.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:13.352 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:19:13.353 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:13.353 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: Deploying daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:19:13.353 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: pgmap v333: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:19:13.353 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:13.353 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:13 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:13.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: Upgrade: Updating mds.foofs.smithi135.brxmmu 2024-04-27T01:19:13.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:13.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi135.brxmmu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-27T01:19:13.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:13.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: Deploying daemon mds.foofs.smithi135.brxmmu on smithi135 2024-04-27T01:19:13.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: pgmap v333: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:19:13.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:13.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:13 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:14.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: osdmap e165: 8 total, 8 up, 8 in 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: Standby daemon mds.foofs.smithi104.wfnrbv assigned to filesystem foofs as rank 0 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T01:19:14.613 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:14 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:replay} 2024-04-27T01:19:14.662 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:19:14.224+0000 7f1c1aafa700 -1 bdev(0x55fb8410c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 4 2024-04-27T01:19:14.662 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:19:14.224+0000 7f1c152ef700 -1 bdev(0x55fb8410c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 6 2024-04-27T01:19:14.662 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:19:14.224+0000 7f1c192f7700 -1 bdev(0x55fb8410c000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 13 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[63921]: 2024-04-27T01:19:14.205+0000 7f8042171700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: osdmap e165: 8 total, 8 up, 8 in 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: Standby daemon mds.foofs.smithi104.wfnrbv assigned to filesystem foofs as rank 0 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-27T01:19:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:14 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:replay} 2024-04-27T01:19:15.557 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:15 smithi135 ceph-mon[50158]: pgmap v334: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:19:15.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:15 smithi104 ceph-mon[63925]: pgmap v334: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 241 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:19:16.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:16 smithi135 ceph-mon[50158]: reconnect by client.14552 172.21.15.104:0/3837531920 after 0 2024-04-27T01:19:16.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:16 smithi135 ceph-mon[50158]: reconnect by client.24337 172.21.15.135:0/3425316284 after 0 2024-04-27T01:19:16.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:16 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:reconnect 2024-04-27T01:19:16.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:16 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:reconnect} 2024-04-27T01:19:16.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:16 smithi104 ceph-mon[63925]: reconnect by client.14552 172.21.15.104:0/3837531920 after 0 2024-04-27T01:19:16.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:16 smithi104 ceph-mon[63925]: reconnect by client.24337 172.21.15.135:0/3425316284 after 0 2024-04-27T01:19:16.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:16 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:reconnect 2024-04-27T01:19:16.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:16 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:reconnect} 2024-04-27T01:19:17.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:17 smithi104 ceph-mon[63925]: pgmap v336: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 316 B/s wr, 2 op/s 2024-04-27T01:19:17.567 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:17 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:rejoin 2024-04-27T01:19:17.567 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:17 smithi104 ceph-mon[63925]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:rejoin} 2024-04-27T01:19:17.567 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:17 smithi104 ceph-mon[63925]: daemon mds.foofs.smithi104.wfnrbv is now active in filesystem foofs as rank 0 2024-04-27T01:19:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:17 smithi135 ceph-mon[50158]: pgmap v336: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 316 B/s wr, 2 op/s 2024-04-27T01:19:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:17 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:rejoin 2024-04-27T01:19:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:17 smithi135 ceph-mon[50158]: fsmap foofs:1/1 {0=foofs.smithi104.wfnrbv=up:rejoin} 2024-04-27T01:19:17.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:17 smithi135 ceph-mon[50158]: daemon mds.foofs.smithi104.wfnrbv is now active in filesystem foofs as rank 0 2024-04-27T01:19:18.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:18 smithi135 ceph-mon[50158]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T01:19:18.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:18 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:active 2024-04-27T01:19:18.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:18 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 2024-04-27T01:19:18.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:18 smithi104 ceph-mon[63925]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T01:19:18.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:18 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618] up:active 2024-04-27T01:19:18.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:18 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 2024-04-27T01:19:19.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:19 smithi135 ceph-mon[50158]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T01:19:19.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:19 smithi104 ceph-mon[63925]: pgmap v337: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 511 B/s wr, 8 op/s 2024-04-27T01:19:21.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:21 smithi135 ceph-mon[50158]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 10 op/s 2024-04-27T01:19:21.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:21 smithi104 ceph-mon[63925]: pgmap v338: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 409 B/s wr, 10 op/s 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: pgmap v339: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: mds.? [v2:172.21.15.135:6832/3288418467,v1:172.21.15.135:6833/3288418467] up:boot 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:19:23.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:23 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:19:23.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: pgmap v339: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-04-27T01:19:23.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:23.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:23.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:23.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-27T01:19:23.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: mds.? [v2:172.21.15.135:6832/3288418467,v1:172.21.15.135:6833/3288418467] up:boot 2024-04-27T01:19:23.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: fsmap foofs:1 {0=foofs.smithi104.wfnrbv=up:active} 1 up:standby 2024-04-27T01:19:23.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:23 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:19:25.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:25 smithi135 ceph-mon[50158]: pgmap v340: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-04-27T01:19:25.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:25 smithi104 ceph-mon[63925]: pgmap v340: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 7.3 KiB/s wr, 14 op/s 2024-04-27T01:19:26.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:26 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:26.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:26 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:26.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:26 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:26.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:26 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:27.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:27 smithi135 ceph-mon[50158]: pgmap v341: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-04-27T01:19:27.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:27.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:27.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:27.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:27 smithi104 ceph-mon[63925]: pgmap v341: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-04-27T01:19:27.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:27.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:27.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:28.416 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (10m) 24s ago 19m 22.8M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (6m) 24s ago 19m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (6m) 3s ago 18m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 24s ago 18m - - 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (29s) 24s ago 13m 13.8M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:19:29.086 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (7s) 3s ago 13m 11.0M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (9m) 24s ago 20m 512M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (8m) 3s ago 18m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (7m) 24s ago 20m 63.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (7m) 3s ago 18m 48.7M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (13m) 24s ago 13m 287M - 4.0 e1d6a67b021e 9c8960b30d53 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (12m) 3s ago 12m 85.9M - 4.0 e1d6a67b021e 80566966bd29 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (9m) 24s ago 18m 21.4M - 1.3.1 1dbe0e931976 66b01d381c5a 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (9m) 3s ago 18m 18.4M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (5m) 24s ago 17m 403M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (5m) 24s ago 17m 382M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (4m) 24s ago 16m 366M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (3m) 24s ago 16m 427M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (2m) 3s ago 16m 352M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:19:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (2m) 3s ago 15m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:19:29.088 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (91s) 3s ago 15m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:19:29.088 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (53s) 3s ago 14m 302M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:19:29.088 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (8m) 24s ago 18m 68.6M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:19:29.489 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:19:29.490 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:19:29.490 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:19:29.490 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:19:29.490 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:19:29.829 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr", 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "mds", 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "osd" 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "16/23 daemons upgraded", 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading mds daemons", 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:19:29.830 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:19:29.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:29 smithi104 ceph-mon[63925]: pgmap v342: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 7.3 KiB/s wr, 11 op/s 2024-04-27T01:19:29.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:29 smithi104 ceph-mon[63925]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:29.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:29 smithi104 ceph-mon[63925]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:29.830 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:29 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1620473103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:29.966 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:29 smithi135 ceph-mon[50158]: pgmap v342: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 7.3 KiB/s wr, 11 op/s 2024-04-27T01:19:29.967 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:29 smithi135 ceph-mon[50158]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:29.967 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:29 smithi135 ceph-mon[50158]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:29.967 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:29 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1620473103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.252 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:19:30.252 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:19:30.253 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi104.wfnrbv"}]': finished 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi135.brxmmu"}]': finished 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.915 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.915 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:30.915 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:30.915 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:30 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2900760026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:19:31.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi104.wfnrbv"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi104.wfnrbv"}]': finished 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi135.brxmmu"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi135.brxmmu"}]': finished 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:31.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:30 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2900760026' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:19:31.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Detected new or changed devices on smithi135 2024-04-27T01:19:31.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: pgmap v343: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-27T01:19:31.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: pgmap v344: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 519 B/s rd, 8.7 KiB/s wr, 4 op/s 2024-04-27T01:19:31.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all mds 2024-04-27T01:19:31.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all rgw 2024-04-27T01:19:31.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all rbd-mirror 2024-04-27T01:19:31.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all ceph-exporter 2024-04-27T01:19:31.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:31 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all iscsi 2024-04-27T01:19:32.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Detected new or changed devices on smithi135 2024-04-27T01:19:32.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: pgmap v343: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.2 KiB/s wr, 6 op/s 2024-04-27T01:19:32.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: pgmap v344: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 519 B/s rd, 8.7 KiB/s wr, 4 op/s 2024-04-27T01:19:32.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all mds 2024-04-27T01:19:32.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all rgw 2024-04-27T01:19:32.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all rbd-mirror 2024-04-27T01:19:32.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all ceph-exporter 2024-04-27T01:19:32.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:31 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all iscsi 2024-04-27T01:19:33.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Upgrade: Updating nfs.foo.0.0.smithi104.yxqnih (1/2) 2024-04-27T01:19:33.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:33.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih 2024-04-27T01:19:33.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: pgmap v345: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Rados config object exists: conf-nfs.foo 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih-rgw 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Bind address in nfs.foo.0.0.smithi104.yxqnih's ganesha conf is defaulting to empty 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:33.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:32 smithi104 ceph-mon[63925]: Deploying daemon nfs.foo.0.0.smithi104.yxqnih on smithi104 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Upgrade: Updating nfs.foo.0.0.smithi104.yxqnih (1/2) 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-27T01:19:33.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: pgmap v345: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Rados config object exists: conf-nfs.foo 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Creating key for client.nfs.foo.0.0.smithi104.yxqnih-rgw 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi104.yxqnih-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Bind address in nfs.foo.0.0.smithi104.yxqnih's ganesha conf is defaulting to empty 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:33.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:32 smithi135 ceph-mon[50158]: Deploying daemon nfs.foo.0.0.smithi104.yxqnih on smithi104 2024-04-27T01:19:35.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:35 smithi104 ceph-mon[63925]: pgmap v346: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-27T01:19:35.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:35 smithi135 ceph-mon[50158]: pgmap v346: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.5 KiB/s wr, 0 op/s 2024-04-27T01:19:37.221 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:37 smithi135 ceph-mon[50158]: pgmap v347: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.6 KiB/s wr, 0 op/s 2024-04-27T01:19:37.221 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:37 smithi135 ceph-mon[50158]: Evicting (and blocklisting) client session 14552 (172.21.15.104:0/3837531920) 2024-04-27T01:19:37.221 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:37 smithi135 ceph-mon[50158]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.104:0/3837531920"}]: dispatch 2024-04-27T01:19:37.221 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:37.222 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:37 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:37.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:37 smithi104 ceph-mon[63925]: pgmap v347: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 1.6 KiB/s wr, 0 op/s 2024-04-27T01:19:37.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:37 smithi104 ceph-mon[63925]: Evicting (and blocklisting) client session 14552 (172.21.15.104:0/3837531920) 2024-04-27T01:19:37.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:37 smithi104 ceph-mon[63925]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.104:0/3837531920"}]: dispatch 2024-04-27T01:19:37.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:37.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:37 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:38.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.104:0/3837531920"}]': finished 2024-04-27T01:19:38.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: osdmap e166: 8 total, 8 up, 8 in 2024-04-27T01:19:38.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: Upgrade: Updating nfs.foo.1.0.smithi135.dbfshs (2/2) 2024-04-27T01:19:38.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:38.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:19:38.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:38 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:38.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.104:0/3837531920"}]': finished 2024-04-27T01:19:38.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: osdmap e166: 8 total, 8 up, 8 in 2024-04-27T01:19:38.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: Upgrade: Updating nfs.foo.1.0.smithi135.dbfshs (2/2) 2024-04-27T01:19:38.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi135.dbfshs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:19:38.413 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:38 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:39.301 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:39 smithi135 ceph-mon[50158]: Rados config object exists: conf-nfs.foo 2024-04-27T01:19:39.301 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:39 smithi135 ceph-mon[50158]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs-rgw 2024-04-27T01:19:39.301 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:39 smithi135 ceph-mon[50158]: Bind address in nfs.foo.1.0.smithi135.dbfshs's ganesha conf is defaulting to empty 2024-04-27T01:19:39.301 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:39 smithi135 ceph-mon[50158]: Deploying daemon nfs.foo.1.0.smithi135.dbfshs on smithi135 2024-04-27T01:19:39.301 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:39 smithi135 ceph-mon[50158]: pgmap v349: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:19:39.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:39 smithi104 ceph-mon[63925]: Rados config object exists: conf-nfs.foo 2024-04-27T01:19:39.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:39 smithi104 ceph-mon[63925]: Creating key for client.nfs.foo.1.0.smithi135.dbfshs-rgw 2024-04-27T01:19:39.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:39 smithi104 ceph-mon[63925]: Bind address in nfs.foo.1.0.smithi135.dbfshs's ganesha conf is defaulting to empty 2024-04-27T01:19:39.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:39 smithi104 ceph-mon[63925]: Deploying daemon nfs.foo.1.0.smithi135.dbfshs on smithi135 2024-04-27T01:19:39.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:39 smithi104 ceph-mon[63925]: pgmap v349: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:19:41.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:41 smithi135 ceph-mon[50158]: pgmap v350: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-27T01:19:41.447 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:41 smithi104 ceph-mon[63925]: pgmap v350: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: Evicting (and blocklisting) client session 24337 (172.21.15.135:0/3425316284) 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.135:0/3425316284"}]: dispatch 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:42.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: Evicting (and blocklisting) client session 24337 (172.21.15.135:0/3425316284) 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.135:0/3425316284"}]: dispatch 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:43.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:43 smithi135 ceph-mon[50158]: pgmap v351: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-27T01:19:43.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:43 smithi135 ceph-mon[50158]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.135:0/3425316284"}]': finished 2024-04-27T01:19:43.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:43 smithi135 ceph-mon[50158]: osdmap e167: 8 total, 8 up, 8 in 2024-04-27T01:19:44.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:43 smithi104 ceph-mon[63925]: pgmap v351: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-27T01:19:44.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:43 smithi104 ceph-mon[63925]: from='mds.? [v2:172.21.15.104:6834/3511308618,v1:172.21.15.104:6835/3511308618]' entity='mds.foofs.smithi104.wfnrbv' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.135:0/3425316284"}]': finished 2024-04-27T01:19:44.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:43 smithi104 ceph-mon[63925]: osdmap e167: 8 total, 8 up, 8 in 2024-04-27T01:19:46.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:45 smithi104 ceph-mon[63925]: pgmap v353: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-27T01:19:46.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:46.106 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:45 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:46.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:45 smithi135 ceph-mon[50158]: pgmap v353: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-27T01:19:46.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:46.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:45 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:47.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:48.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:47 smithi135 ceph-mon[50158]: pgmap v354: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-27T01:19:48.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:48.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:48.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:47 smithi104 ceph-mon[63925]: pgmap v354: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-27T01:19:48.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:48.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:49 smithi135 ceph-mon[50158]: pgmap v355: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:19:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:50.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:49 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:50.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:49 smithi104 ceph-mon[63925]: pgmap v355: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:19:50.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:50.163 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:49 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: pgmap v356: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:19:51.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi104.yxqnih"}]: dispatch 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi104.yxqnih"}]': finished 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi135.dbfshs"}]: dispatch 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi135.dbfshs"}]': finished 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:51.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:51 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: pgmap v356: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:19:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:52.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:52.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi104.yxqnih"}]: dispatch 2024-04-27T01:19:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi104.yxqnih"}]': finished 2024-04-27T01:19:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi135.dbfshs"}]: dispatch 2024-04-27T01:19:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi135.dbfshs"}]': finished 2024-04-27T01:19:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:19:52.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:51 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:19:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:52 smithi135 ceph-mon[50158]: pgmap v357: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 325 B/s rd, 434 B/s wr, 0 op/s 2024-04-27T01:19:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:52 smithi135 ceph-mon[50158]: pgmap v358: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 349 B/s rd, 465 B/s wr, 1 op/s 2024-04-27T01:19:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:52 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all nfs 2024-04-27T01:19:53.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:52 smithi135 ceph-mon[50158]: Upgrade: Setting container_image for all nvmeof 2024-04-27T01:19:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:52 smithi104 ceph-mon[63925]: pgmap v357: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 325 B/s rd, 434 B/s wr, 0 op/s 2024-04-27T01:19:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:52 smithi104 ceph-mon[63925]: pgmap v358: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 349 B/s rd, 465 B/s wr, 1 op/s 2024-04-27T01:19:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:52 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all nfs 2024-04-27T01:19:53.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:52 smithi104 ceph-mon[63925]: Upgrade: Setting container_image for all nvmeof 2024-04-27T01:19:54.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:54 smithi104 ceph-mon[63925]: Upgrade: Updating node-exporter.smithi104 (1/2) 2024-04-27T01:19:54.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:54 smithi104 ceph-mon[63925]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-27T01:19:54.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:54 smithi104 ceph-mon[63925]: pgmap v359: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 868 B/s wr, 2 op/s 2024-04-27T01:19:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:54 smithi135 ceph-mon[50158]: Upgrade: Updating node-exporter.smithi104 (1/2) 2024-04-27T01:19:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:54 smithi135 ceph-mon[50158]: Deploying daemon node-exporter.smithi104 on smithi104 2024-04-27T01:19:55.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:54 smithi135 ceph-mon[50158]: pgmap v359: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 868 B/s wr, 2 op/s 2024-04-27T01:19:57.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:56 smithi135 ceph-mon[50158]: pgmap v360: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 759 B/s wr, 2 op/s 2024-04-27T01:19:57.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:56 smithi104 ceph-mon[63925]: pgmap v360: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 759 B/s wr, 2 op/s 2024-04-27T01:19:58.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:58.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:59.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:58 smithi104 ceph-mon[63925]: pgmap v361: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 434 B/s wr, 1 op/s 2024-04-27T01:19:59.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:58 smithi135 ceph-mon[50158]: pgmap v361: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 434 B/s wr, 1 op/s 2024-04-27T01:20:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:00.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:19:59 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:00.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:00.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:19:59 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:00.611 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:20:01.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:00 smithi104 ceph-mon[63925]: pgmap v362: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 434 B/s wr, 1 op/s 2024-04-27T01:20:01.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:00 smithi104 ceph-mon[63925]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:20:01.205 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:00 smithi135 ceph-mon[50158]: pgmap v362: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 434 B/s wr, 1 op/s 2024-04-27T01:20:01.205 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:00 smithi135 ceph-mon[50158]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (10m) 14s ago 19m 22.9M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (7m) 14s ago 19m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (6m) 16s ago 19m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 14s ago 19m - - 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (62s) 14s ago 13m 36.4M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (39s) 16s ago 13m 12.2M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (9m) 14s ago 20m 513M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (8m) 16s ago 19m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (8m) 14s ago 21m 64.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (8m) 16s ago 19m 49.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (25s) 14s ago 13m 61.3M - 5.7 87e5280d5ae8 ff8851009138 2024-04-27T01:20:01.279 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (20s) 16s ago 13m 23.2M - 5.7 87e5280d5ae8 5cabfa90cdf0 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 starting - - - - 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (9m) 16s ago 18m 17.8M - 1.3.1 1dbe0e931976 c439e7d5588b 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (6m) 14s ago 18m 404M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (5m) 14s ago 17m 382M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (4m) 14s ago 17m 368M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (3m) 14s ago 16m 429M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (3m) 16s ago 16m 353M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (2m) 16s ago 16m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (2m) 16s ago 15m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (86s) 16s ago 15m 303M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:20:01.280 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (8m) 14s ago 18m 68.6M - 2.33.4 514e6a882f6e dac694e91b3a 2024-04-27T01:20:01.688 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:20:01.688 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:20:01.688 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:01.688 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:01.688 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:20:01.689 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "mds", 2024-04-27T01:20:02.035 INFO:teuthology.orchestra.run.smithi104.stdout: "osd", 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout: "nfs" 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "18/23 daemons upgraded", 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:20:02.036 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:20:02.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:01 smithi104 ceph-mon[63925]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:02.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:01 smithi104 ceph-mon[63925]: Upgrade: Updating node-exporter.smithi135 (2/2) 2024-04-27T01:20:02.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:01 smithi104 ceph-mon[63925]: Deploying daemon node-exporter.smithi135 on smithi135 2024-04-27T01:20:02.036 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:01 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/411731042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:02.229 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:01 smithi135 ceph-mon[50158]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:02.230 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:01 smithi135 ceph-mon[50158]: Upgrade: Updating node-exporter.smithi135 (2/2) 2024-04-27T01:20:02.230 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:01 smithi135 ceph-mon[50158]: Deploying daemon node-exporter.smithi135 on smithi135 2024-04-27T01:20:02.230 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:01 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/411731042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:02.438 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:20:02.438 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:20:02.438 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:20:03.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:02 smithi104 ceph-mon[63925]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:03.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:02 smithi104 ceph-mon[63925]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:03.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:02 smithi104 ceph-mon[63925]: pgmap v363: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 409 B/s wr, 1 op/s 2024-04-27T01:20:03.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:02 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/551990749' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:20:03.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:02 smithi135 ceph-mon[50158]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:03.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:02 smithi135 ceph-mon[50158]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:03.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:02 smithi135 ceph-mon[50158]: pgmap v363: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 409 B/s wr, 1 op/s 2024-04-27T01:20:03.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:02 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/551990749' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:20:04.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:03 smithi104 ceph-mon[63925]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:04.264 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:03 smithi135 ceph-mon[50158]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:05.058 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:04 smithi135 ceph-mon[50158]: pgmap v364: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2024-04-27T01:20:05.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:04 smithi104 ceph-mon[63925]: pgmap v364: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 341 B/s wr, 1 op/s 2024-04-27T01:20:06.928 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:06 smithi104 ceph-mon[63925]: pgmap v365: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:06.928 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:06.928 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:06.928 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:06 smithi135 ceph-mon[50158]: pgmap v365: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:09.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:08 smithi135 ceph-mon[50158]: pgmap v366: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:09.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:08 smithi104 ceph-mon[63925]: pgmap v366: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:10.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:09 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:10.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:09 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:10.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:09 smithi104 ceph-mon[63925]: pgmap v367: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:10.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:09 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:10.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:09 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:10.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:09 smithi135 ceph-mon[50158]: pgmap v367: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:11.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:11 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:11.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:11 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:12 smithi104 ceph-mon[63925]: pgmap v368: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:12.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:13.028 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:12 smithi135 ceph-mon[50158]: pgmap v368: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:13.028 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:13.028 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:13.028 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:14.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:14 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:14.663 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:14 smithi104 ceph-mon[63925]: pgmap v369: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:20:14.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:14.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:14 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:14.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:14 smithi135 ceph-mon[50158]: pgmap v369: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:20:16.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: pgmap v370: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:16.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:16.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: pgmap v371: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: pgmap v372: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.863 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.864 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:16 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: pgmap v370: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:16.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: pgmap v371: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: pgmap v372: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.913 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:16.914 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:16 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:18.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:18 smithi104 ceph-mon[63925]: Upgrade: Updating prometheus.smithi104 2024-04-27T01:20:18.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:18 smithi104 ceph-mon[63925]: pgmap v373: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:18.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:18 smithi104 ceph-mon[63925]: Deploying daemon prometheus.smithi104 on smithi104 2024-04-27T01:20:19.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:18 smithi135 ceph-mon[50158]: Upgrade: Updating prometheus.smithi104 2024-04-27T01:20:19.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:18 smithi135 ceph-mon[50158]: pgmap v373: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:19.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:18 smithi135 ceph-mon[50158]: Deploying daemon prometheus.smithi104 on smithi104 2024-04-27T01:20:20.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:20 smithi104 ceph-mon[63925]: pgmap v374: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:21.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:20 smithi135 ceph-mon[50158]: pgmap v374: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:22.827 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:22 smithi104 ceph-mon[63925]: pgmap v375: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:20:23.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:22 smithi135 ceph-mon[50158]: pgmap v375: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:24 smithi104 ceph-mon[63925]: pgmap v376: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:24.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:24 smithi135 ceph-mon[50158]: pgmap v376: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 252 B/s rd, 0 op/s 2024-04-27T01:20:26.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:26 smithi104 ceph-mon[63925]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:20:27.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:26 smithi135 ceph-mon[50158]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:20:27.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:28.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:29.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:28 smithi135 ceph-mon[50158]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:29.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:29.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:28 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:29.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:28 smithi104 ceph-mon[63925]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:29.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:29.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:28 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:30.861 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:30.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:30 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:30.862 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:30 smithi135 ceph-mon[50158]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:30.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:30.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:30 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:30.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:30 smithi104 ceph-mon[63925]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:32.785 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:20:32.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:32 smithi104 ceph-mon[63925]: pgmap v380: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:33.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:32 smithi135 ceph-mon[50158]: pgmap v380: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (11m) 5s ago 20m 23.2M - 0.23.0 ba2b418f427c 87dfe9b105fa 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (7m) 5s ago 20m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (7m) 24s ago 19m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 error 5s ago 20m - - 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (94s) 4s ago 14m 36.5M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:20:33.469 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (71s) 24s ago 14m 12.3M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (10m) 5s ago 21m 516M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (9m) 24s ago 19m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (9m) 5s ago 21m 65.4M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (8m) 24s ago 19m 50.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (57s) 4s ago 14m 62.1M - 5.7 87e5280d5ae8 ff8851009138 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (52s) 24s ago 14m 59.5M - 5.7 87e5280d5ae8 5cabfa90cdf0 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (35s) 4s ago 19m 8526k - 1.5.0 0da6a335fe13 960b4b17bea4 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (28s) 24s ago 19m 5473k - 1.5.0 0da6a335fe13 c45cd3c9fa0d 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (6m) 4s ago 18m 404M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (6m) 4s ago 18m 383M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (5m) 4s ago 17m 369M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (4m) 4s ago 17m 430M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (3m) 24s ago 17m 353M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (3m) 24s ago 16m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (2m) 24s ago 16m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:20:33.470 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (118s) 24s ago 15m 303M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:20:33.471 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (10s) 4s ago 19m 44.4M - 2.43.0 a07b618ecd1d ba0b5492b545 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:20:33.883 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:20:33.884 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:33.884 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.885 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:33.886 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:33 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:34.014 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.015 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.016 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.016 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.016 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:33 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": true, 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [ 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "mon", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "crash", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "mds", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "osd", 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: "nfs" 2024-04-27T01:20:34.229 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:20:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": "18/23 daemons upgraded", 2024-04-27T01:20:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-27T01:20:34.230 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:20:34.230 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:20:34.640 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-27T01:20:34.640 INFO:teuthology.orchestra.run.smithi104.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-27T01:20:34.641 INFO:teuthology.orchestra.run.smithi104.stdout: daemon grafana.smithi104 on smithi104 is in error state 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: pgmap v381: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3088837361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:34.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:34 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1957775090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: pgmap v381: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3088837361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:35.027 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:34 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1957775090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:20:35.853 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:35 smithi104 ceph-mon[63925]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:36.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:35 smithi135 ceph-mon[50158]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:20:36.985 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:36 smithi104 ceph-mon[63925]: Upgrade: Updating alertmanager.smithi104 2024-04-27T01:20:36.985 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:36 smithi104 ceph-mon[63925]: Deploying daemon alertmanager.smithi104 on smithi104 2024-04-27T01:20:36.985 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:36 smithi104 ceph-mon[63925]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:37.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:36 smithi135 ceph-mon[50158]: Upgrade: Updating alertmanager.smithi104 2024-04-27T01:20:37.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:36 smithi135 ceph-mon[50158]: Deploying daemon alertmanager.smithi104 on smithi104 2024-04-27T01:20:37.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:36 smithi135 ceph-mon[50158]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:38.990 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:38 smithi104 ceph-mon[63925]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:39.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:38 smithi135 ceph-mon[50158]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:40.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:40 smithi104 ceph-mon[63925]: pgmap v385: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:40.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:40.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:40.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:40 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:41.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:40 smithi135 ceph-mon[50158]: pgmap v385: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:41.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:41.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:41.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:40 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:43.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:42 smithi135 ceph-mon[50158]: pgmap v386: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:43.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:43.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:42 smithi104 ceph-mon[63925]: pgmap v386: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 213 B/s rd, 0 op/s 2024-04-27T01:20:43.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:44.987 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:44 smithi104 ceph-mon[63925]: pgmap v387: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:20:45.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:44 smithi135 ceph-mon[50158]: pgmap v387: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:20:46.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:46 smithi135 ceph-mon[50158]: pgmap v388: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:46.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:46.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:46 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:46.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:46 smithi104 ceph-mon[63925]: pgmap v388: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:46.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:46.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:46 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:47.567 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:47.568 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:47 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:47.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:47.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:47 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:48.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:48 smithi135 ceph-mon[50158]: pgmap v389: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:48.662 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:48 smithi104 ceph-mon[63925]: pgmap v389: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: pgmap v390: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: pgmap v391: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: pgmap v392: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.089 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104:9093"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.090 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.091 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:50 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: pgmap v390: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: pgmap v391: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-27T01:20:51.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: pgmap v392: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104:9093"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.363 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:51.364 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:50 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:20:52.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:51 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:20:52.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:51 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104:9093"}]: dispatch 2024-04-27T01:20:52.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:51 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:20:52.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:51 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi104:9093"}]: dispatch 2024-04-27T01:20:52.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:52 smithi104 ceph-mon[63925]: pgmap v393: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:20:53.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:52 smithi135 ceph-mon[50158]: pgmap v393: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-27T01:20:53.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:53 smithi104 ceph-mon[63925]: Upgrade: Updating grafana.smithi104 2024-04-27T01:20:53.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:53 smithi104 ceph-mon[63925]: Deploying daemon grafana.smithi104 on smithi104 2024-04-27T01:20:54.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:53 smithi135 ceph-mon[50158]: Upgrade: Updating grafana.smithi104 2024-04-27T01:20:54.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:53 smithi135 ceph-mon[50158]: Deploying daemon grafana.smithi104 on smithi104 2024-04-27T01:20:55.082 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:54 smithi104 ceph-mon[63925]: pgmap v394: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:55.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:54 smithi135 ceph-mon[50158]: pgmap v394: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:57.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:56 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:57.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:56 smithi104 ceph-mon[63925]: pgmap v395: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:57.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:56 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:57.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:56 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:57.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:56 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:57.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:56 smithi135 ceph-mon[50158]: pgmap v395: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:57.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:56 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:20:57.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:56 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:58.490 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:58 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:58.490 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:58 smithi104 ceph-mon[63925]: pgmap v396: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:20:58.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:58 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:58.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:58 smithi135 ceph-mon[50158]: pgmap v396: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2024-04-27T01:21:00.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:20:59 smithi135 ceph-mon[50158]: pgmap v397: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:21:00.411 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:20:59 smithi104 ceph-mon[63925]: pgmap v397: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-27T01:21:02.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:02.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:02 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:02.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:02 smithi104 ceph-mon[63925]: pgmap v398: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:02.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:02.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:02 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:02.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:02 smithi135 ceph-mon[50158]: pgmap v398: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:03.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:03 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:03.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:03 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:03.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:03 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:03.612 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:03 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:04.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:04 smithi104 ceph-mon[63925]: pgmap v399: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:04.513 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:04 smithi135 ceph-mon[50158]: pgmap v399: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:05.052 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (25s) 4s ago 20m 16.2M - 0.25.0 c8568f914cd2 45da2177233f 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (8m) 4s ago 20m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (8m) 56s ago 20m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (10s) 4s ago 20m 48.8M - 9.4.12 2bacad6d85d8 96b09dd94f75 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (2m) 4s ago 14m 36.5M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (103s) 56s ago 14m 12.3M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (10m) 4s ago 22m 522M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (9m) 56s ago 20m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (9m) 4s ago 22m 65.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:21:05.717 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (9m) 56s ago 20m 50.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (90s) 4s ago 14m 63.5M - 5.7 87e5280d5ae8 ff8851009138 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (84s) 56s ago 14m 59.5M - 5.7 87e5280d5ae8 5cabfa90cdf0 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (67s) 4s ago 20m 8140k - 1.5.0 0da6a335fe13 960b4b17bea4 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (60s) 56s ago 19m 5473k - 1.5.0 0da6a335fe13 c45cd3c9fa0d 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (7m) 4s ago 19m 405M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (6m) 4s ago 18m 384M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (5m) 4s ago 18m 369M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (5m) 4s ago 18m 430M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (4m) 56s ago 17m 353M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (3m) 56s ago 17m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (3m) 56s ago 16m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (2m) 56s ago 16m 303M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:21:05.718 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (43s) 4s ago 19m 50.9M - 2.43.0 a07b618ecd1d ba0b5492b545 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:06.161 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:21:06.162 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": null, 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": false, 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "", 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": null, 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:21:06.490 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:21:06.888 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:21:06.889 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:06.889 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: pgmap v400: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104:3000"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104:3000"}]: dispatch 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.890 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.891 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: Upgrade: Finalizing container_image settings 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-27T01:21:06.892 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.893 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: Upgrade: Complete! 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:06.894 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:06.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:06.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:06.895 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:06 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/4011539100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:07.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: pgmap v400: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104:3000"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104:3000"}]: dispatch 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.113 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.114 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: Upgrade: Finalizing container_image settings 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-27T01:21:07.115 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-27T01:21:07.116 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: Upgrade: Complete! 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.117 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:07.118 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:06 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/4011539100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:07 smithi104 ceph-mon[63925]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:07 smithi104 ceph-mon[63925]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:21:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:07 smithi104 ceph-mon[63925]: Cluster is now healthy 2024-04-27T01:21:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:07 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1749486320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:21:08.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:07 smithi135 ceph-mon[50158]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:08.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:07 smithi135 ceph-mon[50158]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-27T01:21:08.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:07 smithi135 ceph-mon[50158]: Cluster is now healthy 2024-04-27T01:21:08.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:07 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1749486320' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:21:09.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:08 smithi104 ceph-mon[63925]: pgmap v401: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:09.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:08 smithi135 ceph-mon[50158]: pgmap v401: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:09.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:11.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:10 smithi104 ceph-mon[63925]: pgmap v402: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:11.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:10 smithi135 ceph-mon[50158]: pgmap v402: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:13.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:12 smithi104 ceph-mon[63925]: pgmap v403: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:13.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:13.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:13.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:12 smithi135 ceph-mon[50158]: pgmap v403: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:13.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:21:13.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:15.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:14 smithi104 ceph-mon[63925]: pgmap v404: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 347 B/s rd, 0 op/s 2024-04-27T01:21:15.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:14 smithi135 ceph-mon[50158]: pgmap v404: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 347 B/s rd, 0 op/s 2024-04-27T01:21:17.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:16 smithi135 ceph-mon[50158]: pgmap v405: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:17.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:16 smithi104 ceph-mon[63925]: pgmap v405: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-04-27T01:21:19.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:18 smithi135 ceph-mon[50158]: pgmap v406: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:19.411 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:18 smithi104 ceph-mon[63925]: pgmap v406: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:21.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:20 smithi135 ceph-mon[50158]: pgmap v407: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:21.411 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:20 smithi104 ceph-mon[63925]: pgmap v407: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:23.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:22 smithi135 ceph-mon[50158]: pgmap v408: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:23.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:22 smithi104 ceph-mon[63925]: pgmap v408: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:24.362 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:23 smithi135 ceph-mon[50158]: pgmap v409: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:24.412 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:23 smithi104 ceph-mon[63925]: pgmap v409: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:27.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:26 smithi135 ceph-mon[50158]: pgmap v410: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:27.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:26 smithi104 ceph-mon[63925]: pgmap v410: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:27.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:28.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:29.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:28 smithi135 ceph-mon[50158]: pgmap v411: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:29.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:28 smithi104 ceph-mon[63925]: pgmap v411: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:31.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:30 smithi135 ceph-mon[50158]: pgmap v412: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:31.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:30 smithi104 ceph-mon[63925]: pgmap v412: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:33.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:32 smithi135 ceph-mon[50158]: pgmap v413: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:33.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:32 smithi104 ceph-mon[63925]: pgmap v413: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:35.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:34 smithi135 ceph-mon[50158]: pgmap v414: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:35.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:34 smithi104 ceph-mon[63925]: pgmap v414: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:37.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:36 smithi135 ceph-mon[50158]: pgmap v415: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:37.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:36 smithi104 ceph-mon[63925]: pgmap v415: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:37.961 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps' 2024-04-27T01:21:38.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:38 smithi104 ceph-mon[63925]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:38.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:38 smithi104 ceph-mon[63925]: pgmap v416: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:39.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:38 smithi135 ceph-mon[50158]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:39.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:38 smithi135 ceph-mon[50158]: pgmap v416: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:39.998 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (60s) 38s ago 21m 16.2M - 0.25.0 c8568f914cd2 45da2177233f 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (8m) 38s ago 21m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (8m) 91s ago 20m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (44s) 38s ago 21m 48.8M - 9.4.12 2bacad6d85d8 96b09dd94f75 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (2m) 38s ago 15m 36.5M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (2m) 91s ago 15m 12.3M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (11m) 38s ago 22m 522M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (10m) 91s ago 20m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (10m) 38s ago 22m 65.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (9m) 91s ago 20m 50.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (2m) 38s ago 15m 63.5M - 5.7 87e5280d5ae8 ff8851009138 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (118s) 91s ago 15m 59.5M - 5.7 87e5280d5ae8 5cabfa90cdf0 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (101s) 38s ago 21m 8140k - 1.5.0 0da6a335fe13 960b4b17bea4 2024-04-27T01:21:39.999 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (94s) 91s ago 20m 5473k - 1.5.0 0da6a335fe13 c45cd3c9fa0d 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (8m) 38s ago 20m 405M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (7m) 38s ago 19m 384M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (6m) 38s ago 19m 369M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (5m) 38s ago 18m 430M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (4m) 91s ago 18m 353M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (4m) 91s ago 17m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (3m) 91s ago 17m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (3m) 91s ago 17m 303M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:21:40.000 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (77s) 38s ago 20m 50.9M - 2.43.0 a07b618ecd1d ba0b5492b545 2024-04-27T01:21:40.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions' 2024-04-27T01:21:40.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:40 smithi104 ceph-mon[63925]: pgmap v417: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:41.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:40 smithi135 ceph-mon[50158]: pgmap v417: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:41.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:41 smithi104 ceph-mon[63925]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:42.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:41 smithi135 ceph-mon[50158]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:42.661 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:21:42.662 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:21:42.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:42 smithi104 ceph-mon[63925]: pgmap v418: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:42.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:42 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3327908322' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:43.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:42 smithi135 ceph-mon[50158]: pgmap v418: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:43.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:43.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:42 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3327908322' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:21:43.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-27T01:21:44.791 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:44 smithi104 ceph-mon[63925]: pgmap v419: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:45.011 INFO:teuthology.orchestra.run.smithi104.stdout:wait for servicemap items w/ changing names to refresh 2024-04-27T01:21:45.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:44 smithi135 ceph-mon[50158]: pgmap v419: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:45.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'sleep 60' 2024-04-27T01:21:46.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:46 smithi104 ceph-mon[63925]: pgmap v420: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:47.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:46 smithi135 ceph-mon[50158]: pgmap v420: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:49.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:48 smithi135 ceph-mon[50158]: pgmap v421: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:49.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:48 smithi104 ceph-mon[63925]: pgmap v421: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:51.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:50 smithi135 ceph-mon[50158]: pgmap v422: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:51.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:50 smithi104 ceph-mon[63925]: pgmap v422: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:53.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:52 smithi135 ceph-mon[50158]: pgmap v423: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:53.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:52 smithi104 ceph-mon[63925]: pgmap v423: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:55.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:54 smithi135 ceph-mon[50158]: pgmap v424: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:55.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:54 smithi104 ceph-mon[63925]: pgmap v424: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:21:57.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:56 smithi135 ceph-mon[50158]: pgmap v425: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:57.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:56 smithi104 ceph-mon[63925]: pgmap v425: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:57.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:58.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:59.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:21:58 smithi135 ceph-mon[50158]: pgmap v426: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:21:59.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:21:58 smithi104 ceph-mon[63925]: pgmap v426: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:01.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:00 smithi135 ceph-mon[50158]: pgmap v427: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:01.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:00 smithi104 ceph-mon[63925]: pgmap v427: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:03.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:02 smithi135 ceph-mon[50158]: pgmap v428: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:03.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:02 smithi104 ceph-mon[63925]: pgmap v428: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:05.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:04 smithi135 ceph-mon[50158]: pgmap v429: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:05.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:04 smithi104 ceph-mon[63925]: pgmap v429: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:07.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:06 smithi135 ceph-mon[50158]: pgmap v430: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:07.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:22:07.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:07.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:06 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:07.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:06 smithi104 ceph-mon[63925]: pgmap v430: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:07.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:22:07.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:07.162 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:06 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:07 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:22:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:07 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:22:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:07 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:07.912 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:07 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:08.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:07 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:22:08.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:07 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:22:08.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:07 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:08.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:07 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:22:09.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:08 smithi135 ceph-mon[50158]: pgmap v431: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:08 smithi104 ceph-mon[63925]: pgmap v431: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:11.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:10 smithi135 ceph-mon[50158]: pgmap v432: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:11.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:10 smithi104 ceph-mon[63925]: pgmap v432: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:13.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:12 smithi135 ceph-mon[50158]: pgmap v433: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:13.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:13.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:12 smithi104 ceph-mon[63925]: pgmap v433: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:13.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:15.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:14 smithi135 ceph-mon[50158]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:15.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:14 smithi104 ceph-mon[63925]: pgmap v434: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:17.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:16 smithi135 ceph-mon[50158]: pgmap v435: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:17.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:16 smithi104 ceph-mon[63925]: pgmap v435: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:19.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:18 smithi135 ceph-mon[50158]: pgmap v436: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:19.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:18 smithi104 ceph-mon[63925]: pgmap v436: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:21.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:20 smithi135 ceph-mon[50158]: pgmap v437: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:21.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:20 smithi104 ceph-mon[63925]: pgmap v437: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:23.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:22 smithi135 ceph-mon[50158]: pgmap v438: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:23.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:22 smithi104 ceph-mon[63925]: pgmap v438: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:25.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:24 smithi135 ceph-mon[50158]: pgmap v439: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:25.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:24 smithi104 ceph-mon[63925]: pgmap v439: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:27.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:26 smithi135 ceph-mon[50158]: pgmap v440: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:27.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:26 smithi104 ceph-mon[63925]: pgmap v440: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:28.112 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:27 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:28.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:27 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:29.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:28 smithi135 ceph-mon[50158]: pgmap v441: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:29.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:28 smithi104 ceph-mon[63925]: pgmap v441: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:31.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:30 smithi135 ceph-mon[50158]: pgmap v442: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:31.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:30 smithi104 ceph-mon[63925]: pgmap v442: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:33.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:32 smithi135 ceph-mon[50158]: pgmap v443: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:33.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:32 smithi104 ceph-mon[63925]: pgmap v443: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:35.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:34 smithi135 ceph-mon[50158]: pgmap v444: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:35.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:34 smithi104 ceph-mon[63925]: pgmap v444: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:37.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:36 smithi135 ceph-mon[50158]: pgmap v445: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:37.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:36 smithi104 ceph-mon[63925]: pgmap v445: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:39.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:38 smithi135 ceph-mon[50158]: pgmap v446: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:39.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:38 smithi104 ceph-mon[63925]: pgmap v446: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:41.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:40 smithi135 ceph-mon[50158]: pgmap v447: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:41.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:40 smithi104 ceph-mon[63925]: pgmap v447: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:43.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:42 smithi135 ceph-mon[50158]: pgmap v448: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:43.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:42 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:43.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:42 smithi104 ceph-mon[63925]: pgmap v448: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:43.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:42 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:45.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:44 smithi135 ceph-mon[50158]: pgmap v449: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:45.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:44 smithi104 ceph-mon[63925]: pgmap v449: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:47.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:46 smithi135 ceph-mon[50158]: pgmap v450: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:47.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:46 smithi104 ceph-mon[63925]: pgmap v450: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:47.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch ps' 2024-04-27T01:22:49.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:48 smithi135 ceph-mon[50158]: pgmap v451: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:49.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:48 smithi104 ceph-mon[63925]: pgmap v451: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:alertmanager.smithi104 smithi104 *:9093,9094 running (2m) 108s ago 22m 16.2M - 0.25.0 c8568f914cd2 45da2177233f 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi104 smithi104 running (9m) 108s ago 22m 6660k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 a8f646bad948 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:crash.smithi135 smithi135 running (9m) 2m ago 21m 6656k - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 d03597364f67 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:grafana.smithi104 smithi104 *:3000 running (114s) 108s ago 22m 48.8M - 9.4.12 2bacad6d85d8 96b09dd94f75 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi104.wfnrbv smithi104 running (3m) 108s ago 16m 36.5M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 22d218109b82 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mds.foofs.smithi135.brxmmu smithi135 running (3m) 2m ago 16m 12.3M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 ba7118695c9b 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi104.dpemzn smithi104 *:8443,9283,8765 running (12m) 108s ago 23m 522M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 52fb1d35daf5 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mgr.smithi135.nwoumo smithi135 *:8443,9283,8765 running (11m) 2m ago 21m 453M - 19.0.0-2447-gb22e2ebd 87e5280d5ae8 c090928172a1 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi104 smithi104 running (11m) 108s ago 23m 65.9M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7822f64f02e2 2024-04-27T01:22:49.912 INFO:teuthology.orchestra.run.smithi104.stdout:mon.smithi135 smithi135 running (10m) 2m ago 21m 50.6M 2048M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 8b38a859eb85 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.0.0.smithi104.yxqnih smithi104 *:2049 running (3m) 108s ago 16m 63.5M - 5.7 87e5280d5ae8 ff8851009138 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:nfs.foo.1.0.smithi135.dbfshs smithi135 *:2049 running (3m) 2m ago 16m 59.5M - 5.7 87e5280d5ae8 5cabfa90cdf0 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi104 smithi104 *:9100 running (2m) 108s ago 22m 8140k - 1.5.0 0da6a335fe13 960b4b17bea4 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:node-exporter.smithi135 smithi135 *:9100 running (2m) 2m ago 21m 5473k - 1.5.0 0da6a335fe13 c45cd3c9fa0d 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.0 smithi104 running (9m) 108s ago 21m 405M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 3914f8babff3 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.1 smithi104 running (8m) 108s ago 20m 384M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 da94f66f5140 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.2 smithi104 running (7m) 108s ago 20m 369M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7b3615fd1d0e 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.3 smithi104 running (6m) 108s ago 19m 430M 1936M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 af5ee718fa5e 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.4 smithi135 running (6m) 2m ago 19m 353M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 60f9178a8c99 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.5 smithi135 running (5m) 2m ago 18m 313M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 bd7bcb98b2f5 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.6 smithi135 running (4m) 2m ago 18m 323M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 7158b89ea83c 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:osd.7 smithi135 running (4m) 2m ago 18m 303M 2704M 19.0.0-2447-gb22e2ebd 87e5280d5ae8 2c21144328a0 2024-04-27T01:22:49.913 INFO:teuthology.orchestra.run.smithi104.stdout:prometheus.smithi104 smithi104 *:9095 running (2m) 108s ago 21m 50.9M - 2.43.0 a07b618ecd1d ba0b5492b545 2024-04-27T01:22:50.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph orch upgrade status' 2024-04-27T01:22:51.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:50 smithi104 ceph-mon[63925]: pgmap v452: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:51.098 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:50 smithi104 ceph-mon[63925]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:22:51.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:50 smithi135 ceph-mon[50158]: pgmap v452: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:51.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:50 smithi135 ceph-mon[50158]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "target_image": null, 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "in_progress": false, 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "which": "", 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "services_complete": [], 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "progress": null, 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "message": "", 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout: "is_paused": false 2024-04-27T01:22:52.632 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:22:52.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:52 smithi104 ceph-mon[63925]: pgmap v453: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:53.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:52 smithi135 ceph-mon[50158]: pgmap v453: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:53.260 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph health detail' 2024-04-27T01:22:54.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:53 smithi135 ceph-mon[50158]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:22:54.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:53 smithi104 ceph-mon[63925]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:22:55.111 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:54 smithi135 ceph-mon[50158]: pgmap v454: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:55.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:54 smithi104 ceph-mon[63925]: pgmap v454: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:22:55.372 INFO:teuthology.orchestra.run.smithi104.stdout:HEALTH_OK 2024-04-27T01:22:55.982 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:55 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/68349763' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:22:56.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions' 2024-04-27T01:22:56.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:55 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/68349763' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-27T01:22:56.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:56 smithi104 ceph-mon[63925]: pgmap v455: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:57.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:56 smithi135 ceph-mon[50158]: pgmap v455: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:58.014 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "mon": { 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "mgr": { 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "osd": { 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 8 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "mds": { 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 2 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "overall": { 2024-04-27T01:22:58.015 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)": 14 2024-04-27T01:22:58.016 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:22:58.016 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-27T01:22:58.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:57 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:58.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:57 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:58.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-27T01:22:58.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:58 smithi104 ceph-mon[63925]: pgmap v456: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:58.997 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:22:58 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1541484908' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:22:59.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:58 smithi135 ceph-mon[50158]: pgmap v456: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:22:59.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:22:58 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1541484908' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:23:00.808 INFO:teuthology.orchestra.run.smithi104.stdout:true 2024-04-27T01:23:01.100 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:00 smithi104 ceph-mon[63925]: pgmap v457: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:01.101 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:00 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/3394685912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:23:01.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:00 smithi135 ceph-mon[50158]: pgmap v457: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:01.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:00 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/3394685912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:23:01.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -e sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-27T01:23:02.911 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:02 smithi104 ceph-mon[63925]: pgmap v458: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:03.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:02 smithi135 ceph-mon[50158]: pgmap v458: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:03.438 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)" 2024-04-27T01:23:04.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:03 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/2931626859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:23:04.029 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-04-27T01:23:04.029 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-04-27T01:23:04.029 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-27T01:23:04.262 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:03 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/2931626859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-27T01:23:05.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:04 smithi104 ceph-mon[63925]: pgmap v459: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:23:05.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:04 smithi135 ceph-mon[50158]: pgmap v459: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-27T01:23:06.019 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-27T01:23:06.019 INFO:teuthology.orchestra.run.smithi104.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-27T00:59:35.829242Z", "last_refresh": "2024-04-27T01:21:01.128512Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-27T00:59:30.302885Z", "last_refresh": "2024-04-27T01:20:08.923676Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-04-27T00:59:33.305126Z", "last_refresh": "2024-04-27T01:21:01.128671Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-27T01:06:01.052571Z", "last_refresh": "2024-04-27T01:20:08.924683Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-27T00:59:28.673051Z", "last_refresh": "2024-04-27T01:20:08.923774Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi104:172.21.15.104=smithi104", "smithi135:172.21.15.135=smithi135"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-27T01:00:19.253264Z", "last_refresh": "2024-04-27T01:20:08.923852Z", "running": 2, "size": 2}}, {"events": ["2024-04-27T01:22:07.358908Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-04-27T01:06:21.976426Z", "last_refresh": "2024-04-27T01:20:08.924825Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-27T00:59:34.600681Z", "last_refresh": "2024-04-27T01:20:08.923973Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6", "last_refresh": "2024-04-27T01:20:08.924116Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-27T00:59:32.016734Z", "last_refresh": "2024-04-27T01:21:01.128831Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-27T01:23:06.692 INFO:tasks.cephadm:nfs.foo has 2/2 2024-04-27T01:23:06.692 DEBUG:teuthology.parallel:result is None 2024-04-27T01:23:06.692 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-27T01:23:06.701 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:23:06.701 DEBUG:teuthology.orchestra.run.smithi104:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-04-27T01:23:06.726 INFO:teuthology.orchestra.run.smithi104.stderr:+ umount /mnt/foo 2024-04-27T01:23:06.738 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T01:23:06.747 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:23:06.747 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-04-27T01:23:07.081 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:06 smithi104 ceph-mon[63925]: pgmap v460: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:07.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:06 smithi135 ceph-mon[50158]: pgmap v460: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:08.155 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:07 smithi104 ceph-mon[63925]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:23:08.155 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:07 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:23:08.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:07 smithi135 ceph-mon[50158]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:23:08.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:07 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:23:08.929 INFO:teuthology.orchestra.run.smithi104.stdout: "foo" 2024-04-27T01:23:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:08 smithi104 ceph-mon[63925]: pgmap v461: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:23:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:23:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:23:09.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:08 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:23:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:08 smithi135 ceph-mon[50158]: pgmap v461: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:23:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:23:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:23:09.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:08 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' 2024-04-27T01:23:09.581 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'ceph nfs export ls foo --detailed' 2024-04-27T01:23:10.159 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:09 smithi104 ceph-mon[63925]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:23:10.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:09 smithi135 ceph-mon[50158]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:23:11.143 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:10 smithi104 ceph-mon[63925]: pgmap v462: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:11.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:10 smithi135 ceph-mon[50158]: pgmap v462: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:11.605 INFO:teuthology.orchestra.run.smithi104.stdout:[ 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: { 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "access_type": "RW", 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "clients": [], 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "cluster_id": "foo", 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "export_id": 1, 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "fsal": { 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "fs_name": "foofs", 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "name": "CEPH", 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "user_id": "nfs.foo.1" 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: }, 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "path": "/", 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: "protocols": [ 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: 4 2024-04-27T01:23:11.606 INFO:teuthology.orchestra.run.smithi104.stdout: ], 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: "pseudo": "/fake", 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: "security_label": true, 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: "squash": "none", 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: "transports": [ 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: "TCP" 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout: } 2024-04-27T01:23:11.607 INFO:teuthology.orchestra.run.smithi104.stdout:] 2024-04-27T01:23:12.219 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'rados -p .nfs --all ls -' 2024-04-27T01:23:13.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:12 smithi104 ceph-mon[63925]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:23:13.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:12 smithi104 ceph-mon[63925]: pgmap v463: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:13.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:12 smithi104 ceph-mon[63925]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:13.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:12 smithi135 ceph-mon[50158]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:23:13.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:12 smithi135 ceph-mon[50158]: pgmap v463: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:23:13.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:12 smithi135 ceph-mon[50158]: from='mgr.24589 172.21.15.104:0/475249749' entity='mgr.smithi104.dpemzn' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:14.071 INFO:teuthology.orchestra.run.smithi104.stdout:foo rec-0000000000000004:nfs.foo.1 2024-04-27T01:23:14.072 INFO:teuthology.orchestra.run.smithi104.stdout:foo grace 2024-04-27T01:23:14.072 INFO:teuthology.orchestra.run.smithi104.stdout:foo rec-0000000000000004:nfs.foo.0 2024-04-27T01:23:14.072 INFO:teuthology.orchestra.run.smithi104.stdout:foo conf-nfs.foo 2024-04-27T01:23:14.072 INFO:teuthology.orchestra.run.smithi104.stdout:foo export-1 2024-04-27T01:23:14.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:13 smithi104 ceph-mon[63925]: mgrmap e43: smithi104.dpemzn(active, since 11m), standbys: smithi135.nwoumo 2024-04-27T01:23:14.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:13 smithi135 ceph-mon[50158]: mgrmap e43: smithi104.dpemzn(active, since 11m), standbys: smithi135.nwoumo 2024-04-27T01:23:14.733 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-27T01:23:14.743 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:23:14.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f5b578-0431-11ef-bc93-c7b262605968 -- bash -c 'set -ex 2024-04-27T01:23:14.743 DEBUG:teuthology.orchestra.run.smithi104:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-04-27T01:23:14.743 DEBUG:teuthology.orchestra.run.smithi104:> ' 2024-04-27T01:23:15.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:14 smithi104 ceph-mon[63925]: pgmap v464: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-27T01:23:15.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:14 smithi135 ceph-mon[50158]: pgmap v464: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-27T01:23:16.559 INFO:teuthology.orchestra.run.smithi104.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-04-27T01:23:16.782 INFO:teuthology.orchestra.run.smithi104.stderr:+ [[ 6 -gt 2 ]] 2024-04-27T01:23:17.123 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:16 smithi104 ceph-mon[63925]: pgmap v465: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T01:23:17.123 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:16 smithi104 ceph-mon[63925]: from='client.? 172.21.15.104:0/1161348161' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-27T01:23:17.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:16 smithi135 ceph-mon[50158]: pgmap v465: 97 pgs: 97 active+clean; 222 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T01:23:17.361 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:16 smithi135 ceph-mon[50158]: from='client.? 172.21.15.104:0/1161348161' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-27T01:23:17.439 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T01:23:17.450 INFO:tasks.cephadm:Teardown begin 2024-04-27T01:23:17.450 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:23:17.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:23:17.507 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T01:23:17.507 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:23:17.531 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:23:17.565 INFO:tasks.cephadm:Stopping all daemons... 2024-04-27T01:23:17.565 INFO:tasks.cephadm.mon.smithi104:Stopping mon.smithi104... 2024-04-27T01:23:17.565 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104 2024-04-27T01:23:17.852 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:17 smithi104 systemd[1]: Stopping Ceph mon.smithi104 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:18.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:17 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[63921]: 2024-04-27T01:23:17.866+0000 7f804797c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi104 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:18.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:17 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104[63921]: 2024-04-27T01:23:17.866+0000 7f804797c700 -1 mon.smithi104@0(leader) e3 *** Got Signal Terminated *** 2024-04-27T01:23:18.161 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:18 smithi104 podman[109431]: 2024-04-27 01:23:18.070260519 +0000 UTC m=+0.300341212 container died 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-27T01:23:18.661 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:18 smithi104 podman[109431]: 2024-04-27 01:23:18.424366707 +0000 UTC m=+0.654447392 container cleanup 7822f64f02e209a41a7f88d890cd30eaffa1e283b22c56d76cd27e90531e8ab2 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:23:18.661 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Apr 27 01:23:18 smithi104 bash[109431]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi104 2024-04-27T01:23:18.836 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi104.service' 2024-04-27T01:23:18.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:18.902 INFO:tasks.cephadm.mon.smithi104:Stopped mon.smithi104 2024-04-27T01:23:18.902 INFO:tasks.cephadm.mon.smithi135:Stopping mon.smithi135... 2024-04-27T01:23:18.902 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi135 2024-04-27T01:23:19.234 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:18 smithi135 systemd[1]: Stopping Ceph mon.smithi135 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:19.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135[50154]: 2024-04-27T01:23:19.247+0000 7f673c772700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi135 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:19.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135[50154]: 2024-04-27T01:23:19.247+0000 7f673c772700 -1 mon.smithi135@1(peon) e3 *** Got Signal Terminated *** 2024-04-27T01:23:19.611 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:19 smithi135 podman[76363]: 2024-04-27 01:23:19.391011703 +0000 UTC m=+0.279304304 container died 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_CLEAN=True) 2024-04-27T01:23:20.011 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:19 smithi135 podman[76363]: 2024-04-27 01:23:19.815527688 +0000 UTC m=+0.703820274 container cleanup 8b38a859eb8532ad95a0fd8955421feb0524f2377f6e11600c60087e64cfad69 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T01:23:20.011 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:Apr 27 01:23:19 smithi135 bash[76363]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-mon-smithi135 2024-04-27T01:23:20.225 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@mon.smithi135.service' 2024-04-27T01:23:20.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:20.273 INFO:tasks.cephadm.mon.smithi135:Stopped mon.smithi135 2024-04-27T01:23:20.273 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-27T01:23:20.273 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.0 2024-04-27T01:23:20.573 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:20 smithi104 systemd[1]: Stopping Ceph osd.0 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:20.911 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:20 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:23:20.586+0000 7f6465089700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:20.911 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:20 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:23:20.586+0000 7f6465089700 -1 osd.0 167 *** Got signal Terminated *** 2024-04-27T01:23:20.911 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:20 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0[69975]: 2024-04-27T01:23:20.586+0000 7f6465089700 -1 osd.0 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:23:26.411 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:26 smithi104 podman[109563]: 2024-04-27 01:23:26.165615682 +0000 UTC m=+5.690626616 container died 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:23:26.823 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:26 smithi104 podman[109563]: 2024-04-27 01:23:26.561101251 +0000 UTC m=+6.086112184 container cleanup 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T01:23:26.823 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:26 smithi104 bash[109563]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0 2024-04-27T01:23:27.128 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:26 smithi104 podman[109579]: 2024-04-27 01:23:26.83766273 +0000 UTC m=+0.666935458 container remove 3914f8babff3695f6cb947bbba9a7c89edeaf88f68422d6277aa89913fff769c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-27T01:23:27.411 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.045934554 +0000 UTC m=+0.018639373 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:23:27.411 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.146659835 +0000 UTC m=+0.119364641 container create d0b597f948febb0b930050b4a8576b8080157c888db992e4c73db66bc7eba1fc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2) 2024-04-27T01:23:27.411 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.366492357 +0000 UTC m=+0.339197163 container init d0b597f948febb0b930050b4a8576b8080157c888db992e4c73db66bc7eba1fc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-27T01:23:27.411 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.370013796 +0000 UTC m=+0.342718621 container start d0b597f948febb0b930050b4a8576b8080157c888db992e4c73db66bc7eba1fc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:23:27.911 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.448836325 +0000 UTC m=+0.421541149 container attach d0b597f948febb0b930050b4a8576b8080157c888db992e4c73db66bc7eba1fc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-27T01:23:27.911 INFO:journalctl@ceph.osd.0.smithi104.stdout:Apr 27 01:23:27 smithi104 podman[109673]: 2024-04-27 01:23:27.662028754 +0000 UTC m=+0.634733575 container died d0b597f948febb0b930050b4a8576b8080157c888db992e4c73db66bc7eba1fc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T01:23:28.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.0.service' 2024-04-27T01:23:28.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:28.319 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-27T01:23:28.320 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-27T01:23:28.320 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.1 2024-04-27T01:23:28.658 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:28 smithi104 systemd[1]: Stopping Ceph osd.1 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:28.911 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:28 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:23:28.671+0000 7f1c36536700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:28.911 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:28 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:23:28.671+0000 7f1c36536700 -1 osd.1 167 *** Got signal Terminated *** 2024-04-27T01:23:28.911 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:28 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1[72528]: 2024-04-27T01:23:28.671+0000 7f1c36536700 -1 osd.1 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:23:34.542 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:34 smithi104 podman[109778]: 2024-04-27 01:23:34.280082693 +0000 UTC m=+5.725559682 container died da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-27T01:23:34.911 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:34 smithi104 podman[109778]: 2024-04-27 01:23:34.654773739 +0000 UTC m=+6.100250727 container cleanup da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-27T01:23:34.911 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:34 smithi104 bash[109778]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1 2024-04-27T01:23:35.304 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:34 smithi104 podman[109795]: 2024-04-27 01:23:34.981194525 +0000 UTC m=+0.702141263 container remove da94f66f5140a7be543c4a1f1c6f3df5a0226798c8db7451f26bdbf61d74701e (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424) 2024-04-27T01:23:35.578 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109889]: 2024-04-27 01:23:35.222555509 +0000 UTC m=+0.018153226 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:23:35.578 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109889]: 2024-04-27 01:23:35.340754638 +0000 UTC m=+0.136352332 container create cffcdf5b44f86261e33b01eb6c696ec87a7521341fb4d11b4aed54f05da05eac (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T01:23:35.578 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109889]: 2024-04-27 01:23:35.592916887 +0000 UTC m=+0.388514581 container init cffcdf5b44f86261e33b01eb6c696ec87a7521341fb4d11b4aed54f05da05eac (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-27T01:23:35.869 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109889]: 2024-04-27 01:23:35.59607296 +0000 UTC m=+0.391670667 container start cffcdf5b44f86261e33b01eb6c696ec87a7521341fb4d11b4aed54f05da05eac (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-27T01:23:35.869 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109889]: 2024-04-27 01:23:35.667994978 +0000 UTC m=+0.463592682 container attach cffcdf5b44f86261e33b01eb6c696ec87a7521341fb4d11b4aed54f05da05eac (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-27T01:23:36.139 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 27 01:23:35 smithi104 podman[109916]: 2024-04-27 01:23:35.907873348 +0000 UTC m=+0.018672902 container died cffcdf5b44f86261e33b01eb6c696ec87a7521341fb4d11b4aed54f05da05eac (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T01:23:36.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.1.service' 2024-04-27T01:23:36.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:36.480 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-27T01:23:36.480 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-27T01:23:36.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.2 2024-04-27T01:23:36.843 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:36 smithi104 systemd[1]: Stopping Ceph osd.2 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:37.161 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:36 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:23:36.856+0000 7fe9586fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:37.161 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:36 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:23:36.856+0000 7fe9586fb700 -1 osd.2 167 *** Got signal Terminated *** 2024-04-27T01:23:37.161 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:36 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2[75227]: 2024-04-27T01:23:36.856+0000 7fe9586fb700 -1 osd.2 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:23:42.560 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:42 smithi104 podman[110027]: 2024-04-27 01:23:42.313254776 +0000 UTC m=+5.591533348 container died 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-27T01:23:42.911 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:42 smithi104 podman[110027]: 2024-04-27 01:23:42.673217039 +0000 UTC m=+5.951495613 container cleanup 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T01:23:42.911 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:42 smithi104 bash[110027]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2 2024-04-27T01:23:43.277 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:42 smithi104 podman[110045]: 2024-04-27 01:23:42.966292034 +0000 UTC m=+0.655453500 container remove 7b3615fd1d0ec4c873edf583f9c2f1f9c9612b9befc8f6aecb38dac356203cd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T01:23:43.587 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.195642802 +0000 UTC m=+0.018368521 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:23:43.587 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.300379143 +0000 UTC m=+0.123104899 container create e34a5f6bf287306bbbdc44aae3cd3e7a90fa1bc38c8ca2dddf3dee0b68431f6b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:23:43.588 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.519314009 +0000 UTC m=+0.342039716 container init e34a5f6bf287306bbbdc44aae3cd3e7a90fa1bc38c8ca2dddf3dee0b68431f6b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T01:23:43.588 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.522805104 +0000 UTC m=+0.345530817 container start e34a5f6bf287306bbbdc44aae3cd3e7a90fa1bc38c8ca2dddf3dee0b68431f6b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-27T01:23:43.588 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.602439298 +0000 UTC m=+0.425165015 container attach e34a5f6bf287306bbbdc44aae3cd3e7a90fa1bc38c8ca2dddf3dee0b68431f6b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T01:23:43.911 INFO:journalctl@ceph.osd.2.smithi104.stdout:Apr 27 01:23:43 smithi104 podman[110136]: 2024-04-27 01:23:43.81350284 +0000 UTC m=+0.636228548 container died e34a5f6bf287306bbbdc44aae3cd3e7a90fa1bc38c8ca2dddf3dee0b68431f6b (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb) 2024-04-27T01:23:44.405 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.2.service' 2024-04-27T01:23:44.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:44.444 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-27T01:23:44.444 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-27T01:23:44.444 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.3 2024-04-27T01:23:44.661 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:44 smithi104 systemd[1]: Stopping Ceph osd.3 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:45.161 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:44 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:23:44.782+0000 7faf5b478700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:45.161 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:44 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:23:44.782+0000 7faf5b478700 -1 osd.3 167 *** Got signal Terminated *** 2024-04-27T01:23:45.161 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:44 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:23:44.782+0000 7faf5b478700 -1 osd.3 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:23:46.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:46 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:46.560+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:47.142 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:46 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:46.904+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:47.797 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:47 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:47.522+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:48.111 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:47 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:47.811+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:48.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:47 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:47.906+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:48.482 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:48 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:48.136+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:48.806 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:48 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:48.495+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:49.111 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:48 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:48.820+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:49.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:48 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:48.951+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:49.451 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:49 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:49.126+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:49.789 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:49 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:49.466+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:50.097 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:49 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:49.803+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:50.098 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:49 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:49.918+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:50.161 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:49 smithi104 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3[77791]: 2024-04-27T01:23:49.780+0000 7faf57a6c700 -1 osd.3 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:24.664173+0000 front 2024-04-27T01:23:24.664318+0000 (oldest deadline 2024-04-27T01:23:49.364389+0000) 2024-04-27T01:23:50.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:50 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:50.111+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:50.661 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:50 smithi104 podman[110244]: 2024-04-27 01:23:50.404580712 +0000 UTC m=+5.725774122 container died af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T01:23:50.794 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:50 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:50.493+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:51.010 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:50 smithi104 podman[110244]: 2024-04-27 01:23:50.840888434 +0000 UTC m=+6.162081852 container cleanup af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:23:51.010 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:50 smithi104 bash[110244]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3 2024-04-27T01:23:51.111 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:50 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:50.808+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:51.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:50 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:50.907+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:51.411 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110261]: 2024-04-27 01:23:51.024730908 +0000 UTC m=+0.618553464 container remove af5ee718fa5ed4fedfe8cefc6da13a85b83681abcbfe2828823eeaa423e989e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:23:51.467 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:51 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:51.141+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:51.775 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:51 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:51.481+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:51.788 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110357]: 2024-04-27 01:23:51.361816764 +0000 UTC m=+0.018391894 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:23:51.788 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110357]: 2024-04-27 01:23:51.484008334 +0000 UTC m=+0.140583438 container create 254e24c06f6109f5afc5610a5379db10463bcb25f4f390ed72aa777141f40b43 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-27T01:23:51.788 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110357]: 2024-04-27 01:23:51.702974351 +0000 UTC m=+0.359549454 container init 254e24c06f6109f5afc5610a5379db10463bcb25f4f390ed72aa777141f40b43 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-27T01:23:51.788 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110357]: 2024-04-27 01:23:51.7062864 +0000 UTC m=+0.362861515 container start 254e24c06f6109f5afc5610a5379db10463bcb25f4f390ed72aa777141f40b43 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:23:51.788 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:51 smithi104 podman[110357]: 2024-04-27 01:23:51.802930289 +0000 UTC m=+0.459505405 container attach 254e24c06f6109f5afc5610a5379db10463bcb25f4f390ed72aa777141f40b43 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424) 2024-04-27T01:23:52.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:51 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:51.942+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:52.111 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:51 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:51.788+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:52.161 INFO:journalctl@ceph.osd.3.smithi104.stdout:Apr 27 01:23:52 smithi104 podman[110357]: 2024-04-27 01:23:52.005656341 +0000 UTC m=+0.662231447 container died 254e24c06f6109f5afc5610a5379db10463bcb25f4f390ed72aa777141f40b43 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-27T01:23:52.469 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:52.150+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:52.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.3.service' 2024-04-27T01:23:52.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:23:52.638 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-27T01:23:52.638 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-27T01:23:52.638 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.4 2024-04-27T01:23:52.795 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:52 smithi135 systemd[1]: Stopping Ceph osd.4 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:23:52.795 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:52.483+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:53.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:52.970+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:53.111 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:52.808+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:53.111 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:52.952+0000 7fcfc6dfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:23:53.111 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:52.952+0000 7fcfc6dfe700 -1 osd.4 167 *** Got signal Terminated *** 2024-04-27T01:23:53.111 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:52 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:52.952+0000 7fcfc6dfe700 -1 osd.4 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:23:53.516 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:53 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:53.134+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:53.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:53 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:53.530+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:53.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:53 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:53.780+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:54.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:54.115+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:54.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:54.001+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:54.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:54.557+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:54.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:54 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:54.795+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:55.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:55.036+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:55.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:55.080+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:55.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:55.080+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.357869+0000 front 2024-04-27T01:23:33.357913+0000 (oldest deadline 2024-04-27T01:23:55.057786+0000) 2024-04-27T01:23:55.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:55.567+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:55.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:55.567+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:23:55.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:55.764+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:55.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:55 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:55.764+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:23:56.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:56.035+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:56.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:56.083+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:56.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:56.083+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.357869+0000 front 2024-04-27T01:23:33.357913+0000 (oldest deadline 2024-04-27T01:23:55.057786+0000) 2024-04-27T01:23:56.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:56.612+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:56.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:56.612+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:23:56.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:56.788+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:56.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:56 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:56.788+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:23:57.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:57.074+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.855596+0000 front 2024-04-27T01:23:21.855490+0000 (oldest deadline 2024-04-27T01:23:47.755449+0000) 2024-04-27T01:23:57.361 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4[55090]: 2024-04-27T01:23:57.074+0000 7fcfc33f2700 -1 osd.4 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.357869+0000 front 2024-04-27T01:23:33.357913+0000 (oldest deadline 2024-04-27T01:23:55.057786+0000) 2024-04-27T01:23:57.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:57.009+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:57.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:57.009+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:23:57.725 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:57.642+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:57.725 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:57.642+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:23:58.021 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:57.739+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:58.021 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:57 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:57.739+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:23:58.346 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:58.035+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:58.347 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:58.035+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:23:58.611 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:58 smithi135 podman[76528]: 2024-04-27 01:23:58.361529383 +0000 UTC m=+5.516138929 container died 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:23:59.005 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:58 smithi135 podman[76528]: 2024-04-27 01:23:58.854012569 +0000 UTC m=+6.008622115 container cleanup 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-27T01:23:59.006 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:58 smithi135 bash[76528]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4 2024-04-27T01:23:59.006 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:58.758+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:59.006 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:58.758+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:23:59.006 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:58.637+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:59.006 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:58 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:58.637+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:23:59.341 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76579]: 2024-04-27 01:23:59.064537434 +0000 UTC m=+0.698580463 container remove 60f9178a8c99a680cff0e8424806d79c13d668a9e3e47f109819ca14ac72119f (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-27T01:23:59.341 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:59.019+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:23:59.342 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:59.019+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:23:59.611 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.258974542 +0000 UTC m=+0.017658481 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:23:59.611 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.357205104 +0000 UTC m=+0.115889027 container create 56690b843289ca4ec5e4868617c9ea2cf52cbe1fc349efab7681e8709d803b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-27T01:23:59.611 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.560398629 +0000 UTC m=+0.319082552 container init 56690b843289ca4ec5e4868617c9ea2cf52cbe1fc349efab7681e8709d803b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-27T01:23:59.611 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.563598227 +0000 UTC m=+0.322282151 container start 56690b843289ca4ec5e4868617c9ea2cf52cbe1fc349efab7681e8709d803b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-27T01:23:59.960 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:59.638+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:23:59.960 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:23:59.638+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:23:59.960 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:59.768+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:23:59.960 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:23:59.768+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:23:59.960 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.641428491 +0000 UTC m=+0.400112418 container attach 56690b843289ca4ec5e4868617c9ea2cf52cbe1fc349efab7681e8709d803b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=squid-b22e2eb) 2024-04-27T01:23:59.961 INFO:journalctl@ceph.osd.4.smithi135.stdout:Apr 27 01:23:59 smithi135 podman[76668]: 2024-04-27 01:23:59.868160911 +0000 UTC m=+0.626844836 container died 56690b843289ca4ec5e4868617c9ea2cf52cbe1fc349efab7681e8709d803b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb) 2024-04-27T01:24:00.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:59.973+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:00.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:23:59 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:23:59.973+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:00.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.4.service' 2024-04-27T01:24:00.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:24:00.547 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-27T01:24:00.548 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-27T01:24:00.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.5 2024-04-27T01:24:00.729 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 systemd[1]: Stopping Ceph osd.5 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:24:00.729 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:00.612+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:00.729 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:00.612+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:00.981 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:00.743+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:00.981 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:00.743+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:00.981 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:00.919+0000 7fbf03e30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:24:00.981 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:00.919+0000 7fbf03e30700 -1 osd.5 167 *** Got signal Terminated *** 2024-04-27T01:24:00.981 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:00.919+0000 7fbf03e30700 -1 osd.5 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:24:01.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:00.995+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:01.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:00 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:00.995+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:01.938 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:01.737+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:01.938 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:01.737+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:01.938 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:01.659+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:01.939 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:01.659+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:02.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:01.952+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:02.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:01 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:01.952+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:02.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:02.747+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:02.861 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:02.747+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:02.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:02.622+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:02.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:02.622+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:03.360 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:02.987+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:03.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:02 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:02.987+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:03.923 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:03.764+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:03.924 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:03.764+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:03.924 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:03.764+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:38.551109+0000 front 2024-04-27T01:23:38.551088+0000 (oldest deadline 2024-04-27T01:24:03.251101+0000) 2024-04-27T01:24:03.924 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:03.671+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:03.924 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:03.671+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:03.924 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:03.671+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:04.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:03.937+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:04.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:03 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:03.937+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:04.742+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:04.743+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:04.743+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:38.551109+0000 front 2024-04-27T01:23:38.551088+0000 (oldest deadline 2024-04-27T01:24:03.251101+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:04.696+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:04.697+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:04.972 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:04.697+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:05.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:04.986+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:05.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:04 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:04.986+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:06.005 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:05.754+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.248960+0000 front 2024-04-27T01:23:22.248858+0000 (oldest deadline 2024-04-27T01:23:47.548850+0000) 2024-04-27T01:24:06.005 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:05.754+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.150179+0000 front 2024-04-27T01:23:32.150134+0000 (oldest deadline 2024-04-27T01:23:55.650072+0000) 2024-04-27T01:24:06.005 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5[57341]: 2024-04-27T01:24:05.754+0000 7fbf00424700 -1 osd.5 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:38.551109+0000 front 2024-04-27T01:23:38.551088+0000 (oldest deadline 2024-04-27T01:24:03.251101+0000) 2024-04-27T01:24:06.005 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:05.723+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:06.005 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:05.723+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:06.006 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:05 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:05.723+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:06.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:06.019+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:06.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:06.019+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:06.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:06.019+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:06.719 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:06 smithi135 podman[76777]: 2024-04-27 01:24:06.481086071 +0000 UTC m=+5.693757216 container died bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T01:24:07.048 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:06 smithi135 podman[76777]: 2024-04-27 01:24:06.853340781 +0000 UTC m=+6.066011935 container cleanup bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T01:24:07.048 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:06 smithi135 bash[76777]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5 2024-04-27T01:24:07.049 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:06.734+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:07.049 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:06.734+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:07.049 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:06 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:06.734+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:07.361 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76805]: 2024-04-27 01:24:07.105162089 +0000 UTC m=+0.627415666 container remove bd7bcb98b2f5450654f19d8a64845ac4cfca315486024e3636d94c122e1435dc (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-27T01:24:07.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:07.062+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:07.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:07.062+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:07.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:07.062+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:07.711 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76894]: 2024-04-27 01:24:07.366279782 +0000 UTC m=+0.017796227 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:24:07.712 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76894]: 2024-04-27 01:24:07.498029777 +0000 UTC m=+0.149546222 container create 19d4ef7c03eb13be4f5eaed8c59bba42570c8d2df2eb4a615d687d7a5b899a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T01:24:07.712 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76894]: 2024-04-27 01:24:07.726273361 +0000 UTC m=+0.377789814 container init 19d4ef7c03eb13be4f5eaed8c59bba42570c8d2df2eb4a615d687d7a5b899a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-27T01:24:08.005 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:07.779+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:08.005 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:07.779+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:08.005 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:07 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:07.779+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:08.006 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76894]: 2024-04-27 01:24:07.729863009 +0000 UTC m=+0.381379457 container start 19d4ef7c03eb13be4f5eaed8c59bba42570c8d2df2eb4a615d687d7a5b899a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-27T01:24:08.006 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:07 smithi135 podman[76894]: 2024-04-27 01:24:07.826147003 +0000 UTC m=+0.477663461 container attach 19d4ef7c03eb13be4f5eaed8c59bba42570c8d2df2eb4a615d687d7a5b899a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb) 2024-04-27T01:24:08.315 INFO:journalctl@ceph.osd.5.smithi135.stdout:Apr 27 01:24:08 smithi135 podman[76894]: 2024-04-27 01:24:08.020549776 +0000 UTC m=+0.672066222 container died 19d4ef7c03eb13be4f5eaed8c59bba42570c8d2df2eb4a615d687d7a5b899a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb) 2024-04-27T01:24:08.316 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.049+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:08.316 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.049+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:08.316 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.049+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:08.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.5.service' 2024-04-27T01:24:08.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:24:08.634 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-27T01:24:08.634 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-27T01:24:08.634 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.6 2024-04-27T01:24:08.861 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 systemd[1]: Stopping Ceph osd.6 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:24:08.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:08.747+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:08.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:08.747+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:08.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:08.747+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.966+0000 7f04cf9aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.966+0000 7f04cf9aa700 -1 osd.6 167 *** Got signal Terminated *** 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:08 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:08.966+0000 7f04cf9aa700 -1 osd.6 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:09.001+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:09.001+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:09.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:09.001+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:10.017 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:09.773+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:10.018 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:09.773+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:10.018 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:09 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:09.773+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:10.360 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.032+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:10.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.032+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:10.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.032+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.993+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.993+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:10.993+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:10.770+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:10.770+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:11.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:10 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:10.770+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:11.995 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:11 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:11.786+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:11.995 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:11 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:11.786+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:11.995 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:11 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:11.786+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:12.360 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:12.009+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:12.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:12.009+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:12.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:12.009+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:13.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:12.806+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:13.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:12.806+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:13.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:12 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:12.806+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:13.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:13.048+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:22.478445+0000 front 2024-04-27T01:23:22.478458+0000 (oldest deadline 2024-04-27T01:23:46.578218+0000) 2024-04-27T01:24:13.112 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:13.048+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:32.279847+0000 front 2024-04-27T01:23:32.280035+0000 (oldest deadline 2024-04-27T01:23:56.980022+0000) 2024-04-27T01:24:13.112 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6[59601]: 2024-04-27T01:24:13.048+0000 7f04cbf9e700 -1 osd.6 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:40.480949+0000 front 2024-04-27T01:23:40.480936+0000 (oldest deadline 2024-04-27T01:24:05.681300+0000) 2024-04-27T01:24:14.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:13.802+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:14.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:13.802+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:14.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:13 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:13.802+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:14.601 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:14 smithi135 podman[77001]: 2024-04-27 01:24:14.360969888 +0000 UTC m=+5.494938518 container died 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-27T01:24:14.861 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:14 smithi135 podman[77001]: 2024-04-27 01:24:14.815651194 +0000 UTC m=+5.949619815 container cleanup 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True) 2024-04-27T01:24:14.861 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:14 smithi135 bash[77001]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6 2024-04-27T01:24:14.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:14 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:14.800+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:14.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:14 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:14.800+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:14.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:14 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:14.800+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:15.361 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77019]: 2024-04-27 01:24:15.084227931 +0000 UTC m=+0.725846729 container remove 7158b89ea83c0f99ebb54912bf1aad547f4e8cec20dc72c2f9aaa1b7a77b3641 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T01:24:15.685 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.418819322 +0000 UTC m=+0.107369138 container create 8d04b119d8a692062f22869dcf26c70adc6a7c5ff693b0b73cd9b2c92ffcaaad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T01:24:15.685 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.329364301 +0000 UTC m=+0.017914118 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:24:15.685 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.613293995 +0000 UTC m=+0.301843812 container init 8d04b119d8a692062f22869dcf26c70adc6a7c5ff693b0b73cd9b2c92ffcaaad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:24:15.685 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.616263379 +0000 UTC m=+0.304813204 container start 8d04b119d8a692062f22869dcf26c70adc6a7c5ff693b0b73cd9b2c92ffcaaad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb) 2024-04-27T01:24:15.685 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.69958544 +0000 UTC m=+0.388135271 container attach 8d04b119d8a692062f22869dcf26c70adc6a7c5ff693b0b73cd9b2c92ffcaaad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-27T01:24:16.111 INFO:journalctl@ceph.osd.6.smithi135.stdout:Apr 27 01:24:15 smithi135 podman[77107]: 2024-04-27 01:24:15.908157547 +0000 UTC m=+0.596707366 container died 8d04b119d8a692062f22869dcf26c70adc6a7c5ff693b0b73cd9b2c92ffcaaad (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:24:16.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:15 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:15.752+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:16.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:15 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:15.752+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:16.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:15 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:15.752+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:16.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:15 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:15.752+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:16.496 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.6.service' 2024-04-27T01:24:16.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:24:16.535 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-27T01:24:16.535 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-27T01:24:16.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.7 2024-04-27T01:24:16.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 systemd[1]: Stopping Ceph osd.7 for 38f5b578-0431-11ef-bc93-c7b262605968... 2024-04-27T01:24:16.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.728+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:16.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.728+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:16.861 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.728+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:16.862 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.728+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:17.361 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.913+0000 7f1791c26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:24:17.361 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.913+0000 7f1791c26700 -1 osd.7 167 *** Got signal Terminated *** 2024-04-27T01:24:17.361 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:16 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:16.913+0000 7f1791c26700 -1 osd.7 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:24:18.108 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:17.776+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:18.108 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:17.776+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:18.108 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:17.776+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:18.108 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:17 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:17.776+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:19.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:18 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:18.803+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:19.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:18 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:18.803+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:19.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:18 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:18.803+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:19.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:18 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:18.803+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:20.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:19.804+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:20.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:19.804+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:20.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:19.804+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:20.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:19 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:19.804+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:21.110 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:20.826+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:21.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:20.826+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:21.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:20.826+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:21.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:20 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:20.826+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:22.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:21 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:21.818+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6806 osd.0 since back 2024-04-27T01:23:21.673423+0000 front 2024-04-27T01:23:21.673482+0000 (oldest deadline 2024-04-27T01:23:46.373324+0000) 2024-04-27T01:24:22.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:21 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:21.818+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6814 osd.1 since back 2024-04-27T01:23:33.275214+0000 front 2024-04-27T01:23:33.275226+0000 (oldest deadline 2024-04-27T01:23:54.975096+0000) 2024-04-27T01:24:22.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:21 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:21.818+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6822 osd.2 since back 2024-04-27T01:23:36.676275+0000 front 2024-04-27T01:23:36.676338+0000 (oldest deadline 2024-04-27T01:24:03.076521+0000) 2024-04-27T01:24:22.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:21 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:21.818+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.104:6830 osd.3 since back 2024-04-27T01:23:49.478061+0000 front 2024-04-27T01:23:49.478036+0000 (oldest deadline 2024-04-27T01:24:15.378021+0000) 2024-04-27T01:24:22.111 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:21 smithi135 ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7[61862]: 2024-04-27T01:24:21.818+0000 7f178a212700 -1 osd.7 167 heartbeat_check: no reply from 172.21.15.135:6804 osd.4 since back 2024-04-27T01:23:57.078903+0000 front 2024-04-27T01:23:57.078975+0000 (oldest deadline 2024-04-27T01:24:21.178899+0000) 2024-04-27T01:24:22.778 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:22 smithi135 podman[77244]: 2024-04-27 01:24:22.400486165 +0000 UTC m=+5.622795054 container died 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, ceph=True) 2024-04-27T01:24:23.089 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:22 smithi135 podman[77244]: 2024-04-27 01:24:22.793572758 +0000 UTC m=+6.015881655 container cleanup 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:24:23.089 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:22 smithi135 bash[77244]: ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7 2024-04-27T01:24:23.361 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77274]: 2024-04-27 01:24:23.103881257 +0000 UTC m=+0.702232812 container remove 2c21144328a08074494baee7513c1a2be6087e192b8fd991e98451218c20407c (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True, CEPH_POINT_RELEASE=) 2024-04-27T01:24:23.701 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77361]: 2024-04-27 01:24:23.362085019 +0000 UTC m=+0.017857570 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:24:23.701 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77361]: 2024-04-27 01:24:23.496525056 +0000 UTC m=+0.152297601 container create 3c75afbfd462b4d798f650c408581e1b4c5e1259b5fbedb4c2fde9c78f7d3886 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:24:23.701 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77361]: 2024-04-27 01:24:23.716444432 +0000 UTC m=+0.372216965 container init 3c75afbfd462b4d798f650c408581e1b4c5e1259b5fbedb4c2fde9c78f7d3886 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-27T01:24:23.997 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77361]: 2024-04-27 01:24:23.71944315 +0000 UTC m=+0.375215686 container start 3c75afbfd462b4d798f650c408581e1b4c5e1259b5fbedb4c2fde9c78f7d3886 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb) 2024-04-27T01:24:23.997 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:23 smithi135 podman[77361]: 2024-04-27 01:24:23.791737418 +0000 UTC m=+0.447509953 container attach 3c75afbfd462b4d798f650c408581e1b4c5e1259b5fbedb4c2fde9c78f7d3886 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-27T01:24:24.280 INFO:journalctl@ceph.osd.7.smithi135.stdout:Apr 27 01:24:24 smithi135 podman[77361]: 2024-04-27 01:24:24.012271061 +0000 UTC m=+0.668043600 container died 3c75afbfd462b4d798f650c408581e1b4c5e1259b5fbedb4c2fde9c78f7d3886 (image=quay.ceph.io/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-38f5b578-0431-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T01:24:24.746 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38f5b578-0431-11ef-bc93-c7b262605968@osd.7.service' 2024-04-27T01:24:24.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:24:24.783 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-27T01:24:24.783 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38f5b578-0431-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T01:27:19.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38f5b578-0431-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T01:27:38.854 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:27:38.887 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:27:38.912 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T01:27:38.914 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi104/crash 2024-04-27T01:27:38.915 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash -- . 2024-04-27T01:27:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T01:27:38.976 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-04-27T01:27:38.978 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi135/crash 2024-04-27T01:27:38.979 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash -- . 2024-04-27T01:27:39.024 INFO:teuthology.orchestra.run.smithi135.stderr:tar: /var/lib/ceph/38f5b578-0431-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T01:27:39.024 INFO:teuthology.orchestra.run.smithi135.stderr:tar: Error is not recoverable: exiting now 2024-04-27T01:27:39.025 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T01:27:39.025 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T01:27:39.053 INFO:teuthology.orchestra.run.smithi104.stdout:2024-04-27T01:10:35.997918+0000 mon.smithi104 (mon.0) 859 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:27:39.053 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-27T01:27:39.054 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[SEC\]' /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T01:27:39.120 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]' /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T01:27:39.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[WRN\]' /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-27T01:27:39.254 INFO:teuthology.orchestra.run.smithi104.stdout:2024-04-27T01:10:35.997918+0000 mon.smithi104 (mon.0) 859 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-27T01:27:39.255 INFO:tasks.cephadm:Compressing logs... 2024-04-27T01:27:39.255 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T01:27:39.297 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T01:27:39.318 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T01:27:39.318 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-04-27T01:27:39.318 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log 2024-04-27T01:27:39.318 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T01:27:39.320 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T01:27:39.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-27T01:27:39.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T01:27:39.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T01:27:39.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T01:27:39.321 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi104.dpemzn.log 2024-04-27T01:27:39.324 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.5% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T01:27:39.324 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T01:27:39.324 INFO:teuthology.orchestra.run.smithi104.stderr: 87.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T01:27:39.324 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi104.dpemzn.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi104.wfnrbv.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:: No such file or directory 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi135.nwoumo.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:gzip/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi135.nwoumo.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-27T01:27:39.326 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-27T01:27:39.327 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-27T01:27:39.327 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-27T01:27:39.327 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T01:27:39.327 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.6.log: 87.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T01:27:39.327 INFO:teuthology.orchestra.run.smithi135.stderr: 91.7% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T01:27:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log: 91.5% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T01:27:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi104.log 2024-04-27T01:27:39.329 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi104.wfnrbv.log: 89.5% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi104.wfnrbv.log.gz 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi135.brxmmu.log 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi135.log 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi135.brxmmu.log: gzip -5 --verbose -- /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr: 85.5% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T01:27:39.331 INFO:teuthology.orchestra.run.smithi135.stderr: 87.7% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mds.foofs.smithi135.brxmmu.log.gz 2024-04-27T01:27:39.340 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi104.log: 91.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T01:27:39.344 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi135.log: /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log: 91.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T01:27:39.354 INFO:teuthology.orchestra.run.smithi104.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T01:27:39.392 INFO:teuthology.orchestra.run.smithi135.stderr: 89.7% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi135.nwoumo.log.gz 2024-04-27T01:27:39.600 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mgr.smithi104.dpemzn.log.gz 2024-04-27T01:27:39.654 INFO:teuthology.orchestra.run.smithi135.stderr: 92.6% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi135.log.gz 2024-04-27T01:27:41.375 INFO:teuthology.orchestra.run.smithi104.stderr: 91.9% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-mon.smithi104.log.gz 2024-04-27T01:27:53.075 INFO:teuthology.orchestra.run.smithi104.stderr: 94.5% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T01:27:54.716 INFO:teuthology.orchestra.run.smithi135.stderr: 93.9% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-27T01:27:56.244 INFO:teuthology.orchestra.run.smithi104.stderr: 94.1% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T01:27:58.378 INFO:teuthology.orchestra.run.smithi135.stderr: 94.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-27T01:27:59.137 INFO:teuthology.orchestra.run.smithi104.stderr: 94.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T01:27:59.642 INFO:teuthology.orchestra.run.smithi104.stderr: 94.1% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-27T01:27:59.644 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-27T01:27:59.644 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m20.335s 2024-04-27T01:27:59.644 INFO:teuthology.orchestra.run.smithi104.stderr:user 1m10.351s 2024-04-27T01:27:59.644 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m2.745s 2024-04-27T01:27:59.930 INFO:teuthology.orchestra.run.smithi135.stderr: 94.2% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-27T01:28:02.927 INFO:teuthology.orchestra.run.smithi135.stderr: 94.1% -- replaced with /var/log/ceph/38f5b578-0431-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-27T01:28:02.929 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-04-27T01:28:02.929 INFO:teuthology.orchestra.run.smithi135.stderr:real 0m23.618s 2024-04-27T01:28:02.929 INFO:teuthology.orchestra.run.smithi135.stderr:user 1m15.951s 2024-04-27T01:28:02.929 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m3.015s 2024-04-27T01:28:02.930 INFO:tasks.cephadm:Archiving logs... 2024-04-27T01:28:02.930 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi104/log 2024-04-27T01:28:02.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T01:28:12.793 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi135/log 2024-04-27T01:28:12.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T01:28:23.400 INFO:tasks.cephadm:Removing cluster... 2024-04-27T01:28:23.400 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38f5b578-0431-11ef-bc93-c7b262605968 --force 2024-04-27T01:28:23.774 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38f5b578-0431-11ef-bc93-c7b262605968 --force 2024-04-27T01:28:24.156 INFO:tasks.cephadm:Removing cephadm ... 2024-04-27T01:28:24.156 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T01:28:24.171 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T01:28:24.185 INFO:tasks.cephadm:Teardown complete 2024-04-27T01:28:24.186 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T01:28:24.196 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T01:28:24.196 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:28:24.213 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:28:24.225 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 59 +1202us[+1202us] +/- 64ms 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -1738us[-1690us] +/- 59ms 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +1084us[+1084us] +/- 69ms 2024-04-27T01:28:24.230 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:28:24.239 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:28:24.243 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:28:24.244 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-04-27T01:28:24.244 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 61 +1164us[+1196us] +/- 64ms 2024-04-27T01:28:24.244 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1774us[-1743us] +/- 59ms 2024-04-27T01:28:24.244 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 122 +1047us[+1078us] +/- 68ms 2024-04-27T01:28:24.244 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:28:24.244 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T01:28:24.256 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T01:28:24.257 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T01:28:24.270 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T01:28:24.347 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T01:28:24.417 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:28:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:28:24.444 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T01:28:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:28:24.471 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T01:28:24.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T01:28:24.492 INFO:teuthology.task.internal:Duration was 2026.277434 seconds 2024-04-27T01:28:24.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T01:28:24.504 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T01:28:24.504 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T01:28:24.506 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T01:28:24.540 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:28:24.546 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:28:24.896 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T01:28:24.897 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-04-27T01:28:24.897 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T01:28:24.916 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-04-27T01:28:24.917 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T01:28:24.960 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T01:28:24.960 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T01:28:24.963 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T01:28:25.025 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T01:28:25.026 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T01:28:25.028 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T01:28:38.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T01:28:38.814 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T01:28:38.814 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T01:28:38.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T01:28:38.866 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T01:28:38.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:28:38.880 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:28:38.900 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-04-27T01:28:38.932 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-04-27T01:28:39.224 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:28:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:28:39.239 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:28:39.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:28:39.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T01:28:39.293 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T01:28:39.293 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi104 2024-04-27T01:28:39.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T01:28:39.351 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445/remote/smithi135 2024-04-27T01:28:39.352 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T01:28:39.395 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T01:28:39.395 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T01:28:39.397 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T01:28:39.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T01:28:39.456 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T01:28:39.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T01:28:39.467 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T01:28:39.467 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T01:28:39.469 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T01:28:39.480 INFO:teuthology.orchestra.run.smithi104.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 01:28 /home/ubuntu/cephtest 2024-04-27T01:28:39.501 INFO:teuthology.orchestra.run.smithi135.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 01:28 /home/ubuntu/cephtest 2024-04-27T01:28:39.502 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T01:28:39.513 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T01:28:39.535 INFO:teuthology.nuke:Checking targets against current locks 2024-04-27T01:28:39.558 DEBUG:teuthology.nuke:shortname: smithi104 2024-04-27T01:28:39.558 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T01:28:39.580 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:30.241081', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDluEo2n692/ghh2W9N94ppXOknaEDw7tvLcNNAkzp7idm2zlPynFhTEVQA8qwGuTZQCBDnFXUDi8O7FWFpu0Zk='} 2024-04-27T01:28:39.581 DEBUG:teuthology.nuke:shortname: smithi135 2024-04-27T01:28:39.581 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T01:28:39.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675445', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:30.239713', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH++j15LfKWyud1/PZY3bWcv3bovYDD196xqx7r8LBDV9pXC1tWjU8KO73ZIdT9FT5blwPmDcPrBp4MmxCLFWmM='} 2024-04-27T01:28:39.628 INFO:teuthology.orchestra.console.smithi104:Power off 2024-04-27T01:28:39.629 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T01:28:39.653 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-04-27T01:28:39.653 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:39.668 INFO:teuthology.orchestra.console.smithi135:Power off 2024-04-27T01:28:39.669 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T01:28:39.677 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-27T01:28:39.692 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2024-04-27T01:28:39.693 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:39.716 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-04-27T01:28:43.678 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:43.717 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:43.789 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-27T01:28:43.827 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-04-27T01:28:47.793 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:47.827 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:47.906 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-04-27T01:28:47.937 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2024-04-27T01:28:48.037 INFO:teuthology.orchestra.console.smithi135:Power off completed 2024-04-27T01:28:51.910 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T01:28:52.019 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-04-27T01:28:52.120 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-04-27T01:28:52.224 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2026.2774341106415 failure_reason: '"2024-04-27T01:10:35.997918+0000 mon.smithi104 (mon.0) 859 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-27T01:28:52.224 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T01:28:52.293 INFO:teuthology.run:FAIL